libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17216:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17216:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17217:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17217:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17218:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17218:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17219:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17219:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17220:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17220:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17221:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17221:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17222:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17222:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17223:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17223:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17224:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17224:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17225:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17225:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17226:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17226:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17227:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17227:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17228:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17228:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17229:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17229:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17231:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17231:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17232:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17232:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17233:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17233:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17234:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17234:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17235:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17235:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17236:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17236:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17237:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17237:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17238:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17238:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17239:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17239:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17240:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17240:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17241:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17241:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17242:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17242:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17243:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17243:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17244:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17244:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17245:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17245:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17246:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17246:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17247:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17247:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17248:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17248:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17249:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17249:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17250:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17250:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17251:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17251:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17252:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17252:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17253:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17253:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17254:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17254:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17255:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17255:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17256:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17256:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17257:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17257:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17258:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17258:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17258:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17259:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17259:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17260:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17260:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17261:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17261:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17262:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17262:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17263:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17263:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17264:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17264:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17265:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17265:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17266:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17266:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17267:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17267:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17268:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17268:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17269:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17269:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17270:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17270:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17271:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17271:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17272:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17272:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17273:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17273:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17274:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17274:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17275:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17275:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17276:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17276:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17277:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17277:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17278:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17278:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17279:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17279:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17280:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17280:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17281:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17281:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17282:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17282:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17282:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17283:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17283:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17284:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17284:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17285:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17285:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17286:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17286:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17287:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17287:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17288:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17288:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17289:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17289:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17290:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17290:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17290:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17291:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17291:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17292:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17292:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17292:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17293:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17293:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17294:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17294:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17295:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17295:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17296:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17296:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17297:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17297:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17298:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17298:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17299:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17299:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17300:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17300:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17301:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17301:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17302:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17302:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17303:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17303:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17304:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17304:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17304:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17305:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17305:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17306:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17306:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17306:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17307:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17307:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17308:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17308:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17309:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17309:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17310:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17310:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17311:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17311:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17312:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17312:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17313:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17313:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17314:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17314:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17315:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17315:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17315:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17316:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17316:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17317:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17317:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17318:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17318:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17319:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17319:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17320:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17320:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17321:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17321:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17322:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17322:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17322:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17323:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17323:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17323:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17325:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17325:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17326:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17326:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17327:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17327:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17328:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17328:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17329:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17329:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17330:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17330:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17331:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17331:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17332:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17332:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17333:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17333:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17334:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17334:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17335:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17335:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17335:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17336:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17336:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17336:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17337:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17337:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17338:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17338:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17339:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17339:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17340:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17340:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17341:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17341:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17342:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17342:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17342:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17343:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17343:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17344:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17344:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17224:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17229:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17230:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17230:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17257:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17269:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17328:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17280:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17291:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17308:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17288:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17309:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17314:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17330:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17264:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17297:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17296:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17223:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17300:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17250:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17319:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17339:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17279:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17234:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17236:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17278:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17318:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17270:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17275:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17294:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17298:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17239:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17331:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17283:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17237:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17242:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17281:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17225:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17262:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17246:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17284:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17245:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17253:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17219:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17255:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17301:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17247:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17221:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17276:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17305:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17293:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17325:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17333:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17343:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17233:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17241:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17249:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17263:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17286:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17310:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17312:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17272:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17332:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17248:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17337:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17344:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17218:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17228:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17226:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17303:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17326:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17243:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17338:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17320:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17321:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17235:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17329:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17238:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17259:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17341:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17251:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17295:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17277:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17256:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17220:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17285:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17307:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17289:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17327:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17222:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17265:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17287:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17240:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17334:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17340:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17261:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17311:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17267:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17299:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17266:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17227:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17302:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17317:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17260:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17313:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17216:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17232:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17274:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17254:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17244:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17217:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17268:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17271:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17273:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17231:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17316:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17252:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17230:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85883:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85883:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85972:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85972:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85989:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85989:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85913:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85913:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85901:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85901:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85904:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85904:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85909:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85909:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85935:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85935:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85934:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85934:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85958:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85958:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85959:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85959:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85929:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85929:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85932:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85932:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85962:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85962:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85893:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85893:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85927:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85927:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85978:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85978:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85980:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85980:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85990:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85990:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85906:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85906:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85908:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85908:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85933:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85933:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85936:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85936:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85951:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85951:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85968:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85968:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85976:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85976:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85870:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85870:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85876:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85876:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85894:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85894:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85905:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85905:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85924:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85924:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85930:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85930:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85941:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85941:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85942:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85942:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85950:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85950:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85953:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85953:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85955:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85955:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85979:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85979:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85984:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85984:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85881:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85881:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85969:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85969:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85880:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85880:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85900:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85900:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85916:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85916:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85983:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85983:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85961:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85961:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85954:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85954:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85868:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85868:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85879:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85879:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85865:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85865:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85922:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85922:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85875:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85875:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85892:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85892:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85943:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85943:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85945:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85945:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85946:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85946:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85948:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85948:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85885:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85885:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85887:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85887:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85888:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85888:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85896:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85896:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85918:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85918:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85919:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85919:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85920:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85920:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85939:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85939:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85986:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85986:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85873:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85873:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85874:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85874:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85878:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85878:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85898:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85898:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85915:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85915:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85923:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85923:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85926:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85926:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85957:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85957:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85966:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85966:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85971:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85971:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85991:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85991:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85940:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85940:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85938:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85938:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85965:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85965:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85866:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85866:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85867:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85867:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85869:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85869:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85871:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85871:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85872:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85872:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85877:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85877:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85882:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85882:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85883:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85884:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85884:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85886:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85886:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85889:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85889:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85890:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85890:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85891:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85891:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85895:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85895:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85897:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85897:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85899:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85899:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85901:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85902:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85902:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85903:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85903:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85904:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85907:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85907:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85909:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85910:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85910:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85911:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85911:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85912:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85912:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85913:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85914:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85914:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85917:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85917:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85921:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85921:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85925:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85925:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85928:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85928:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85929:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85931:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85931:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85934:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85935:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85937:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85937:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85944:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85944:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85947:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85947:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85949:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85949:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85952:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85952:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85956:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85956:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85958:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85959:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85960:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85960:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85962:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85963:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85963:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85964:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85964:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85967:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85967:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85970:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85970:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85972:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85973:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85973:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85974:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85974:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85975:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85975:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85977:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85977:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85981:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85981:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85982:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85982:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85985:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85985:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85987:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85987:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85988:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85988:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85989:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85992:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85992:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85932:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85978:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85980:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85990:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85893:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85927:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85933:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85906:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85908:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85936:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85968:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85976:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85951:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85955:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85870:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85876:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85924:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85950:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85953:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85984:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85930:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85941:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85979:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85905:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85942:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85894:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85881:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85969:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85880:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85900:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85916:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85983:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85961:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85868:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85879:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85922:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85954:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85865:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85945:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85948:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85946:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85892:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85875:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85939:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85943:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85919:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85887:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85888:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85918:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85920:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85986:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85915:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85957:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85878:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85966:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85991:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85885:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85926:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85898:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85923:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85971:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85896:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85874:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85873:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85940:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85938:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85965:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85882:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85992:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85970:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85872:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85925:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85866:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85897:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85982:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85956:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85867:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85899:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85907:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85884:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85891:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85871:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85912:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85960:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85963:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85975:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85949:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85914:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85890:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85895:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85985:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85952:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85889:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85877:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85910:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85903:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85967:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85917:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85921:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85886:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85902:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85931:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85981:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85944:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85977:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85928:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85973:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85974:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85937:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85964:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85947:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85869:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85911:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85987:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85988:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17328:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17281:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17219:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17216:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17220:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17329:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17223:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17236:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17333:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17316:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17325:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17259:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17265:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17272:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17327:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17318:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17251:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17299:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85877:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85877:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85889:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85889:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17224:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85907:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85907:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85949:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85949:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85897:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85897:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17231:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85877:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17232:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85963:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85963:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85889:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85888:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85888:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85907:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17240:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17242:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85920:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85920:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85949:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85903:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85903:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85897:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17248:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17252:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85895:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85895:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85963:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17254:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85975:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85975:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17256:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85956:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85956:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85888:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17263:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85982:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85982:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17266:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85887:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85887:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17267:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85920:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85893:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85893:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85903:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85876:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85876:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17280:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85890:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85890:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17283:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85986:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85986:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17288:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17289:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85880:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85880:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85935:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85935:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85895:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85967:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85967:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85975:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85866:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85866:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17303:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85940:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85940:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17304:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17305:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85936:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85936:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17309:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85913:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85913:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17311:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85956:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85941:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85941:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85982:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85925:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85925:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85887:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85924:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85924:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85984:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85984:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85960:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85960:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85893:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85938:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85938:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85923:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85923:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85930:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85930:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17253:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85928:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85928:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85876:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85898:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85898:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85890:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85899:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85899:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85912:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85912:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85916:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85916:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85946:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85946:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85969:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85969:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85881:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85881:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85945:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85945:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85986:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85880:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85992:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85992:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85935:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85967:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17217:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17218:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17221:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17222:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17225:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17226:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17227:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17228:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17229:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17230:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17233:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17234:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17235:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17237:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17238:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17239:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17241:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17243:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17244:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17245:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17246:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17247:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17249:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17250:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17255:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17257:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17258:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17260:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17261:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17262:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17264:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17268:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17269:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17270:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17271:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17273:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17274:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17275:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17276:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17277:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17278:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17279:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17282:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17284:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17285:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17286:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17287:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17290:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17291:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17292:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17293:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17294:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17295:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17296:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17297:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17298:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17300:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17301:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17302:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17306:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17307:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17308:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17310:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17312:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17313:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17314:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17315:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17317:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17319:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17320:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17321:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17322:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17323:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17326:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17330:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17331:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17332:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17334:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17335:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17336:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17337:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17338:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17339:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17340:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17341:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17342:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17343:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:17344:core:mr:ofi_default_cache_size():78 default cache size=2110846816 libfabric:85866:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85869:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85869:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85870:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85870:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85871:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85871:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85872:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85872:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85873:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85873:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85875:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85875:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85878:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85878:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85879:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85879:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85881:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85886:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85886:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85891:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85891:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85894:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85894:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85896:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85896:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85898:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85899:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85900:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85900:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85901:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85901:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85902:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85902:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85904:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85904:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85905:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85905:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85908:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85908:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85910:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85910:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85912:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85913:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85914:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85914:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85915:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85915:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85916:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85917:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85917:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85923:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85924:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85925:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85927:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85927:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85928:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85929:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85929:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85930:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85932:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85932:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85933:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85933:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85934:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85934:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85936:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85937:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85937:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85938:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85939:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85939:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85940:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85941:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85942:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85942:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85944:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85944:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85945:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85946:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85948:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85948:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85950:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85950:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85951:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85951:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85952:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85952:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85953:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85953:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85954:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85954:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85955:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85955:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85957:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85957:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85958:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85958:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85960:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85961:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85961:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85965:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85965:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85966:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85966:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85969:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85970:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85970:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85972:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85972:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85981:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85981:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85983:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85983:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85984:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85985:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85985:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85987:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85987:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85989:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85989:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85990:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85990:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85991:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85991:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85992:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85918:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85918:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85968:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85968:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85988:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85988:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85882:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85882:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85931:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85931:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85868:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85868:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85906:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85906:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85973:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85973:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85991:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85971:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85971:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85865:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85865:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85867:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85867:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85868:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85869:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85870:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85871:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85872:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85873:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85874:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85874:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85875:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85878:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85879:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85882:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85883:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85883:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85884:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85884:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85885:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85885:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85886:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85891:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85892:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85892:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85894:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85896:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85900:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85901:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85902:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85904:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85905:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85906:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85908:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85909:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85909:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85910:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85911:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85911:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85911:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85914:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85915:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85917:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85918:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85919:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85919:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85921:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85921:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85922:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85922:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85926:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85926:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85927:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85929:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85931:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85932:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85933:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85934:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85937:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85939:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85942:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85943:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85943:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85944:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85947:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85947:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85948:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85950:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85951:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85952:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85953:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85954:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85955:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85957:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85958:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85959:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85959:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85961:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85962:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85962:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85962:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85964:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85964:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85964:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85965:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85966:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85968:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85970:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85971:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85972:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85973:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85974:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85974:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85976:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85976:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85977:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85977:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85977:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85978:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85978:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85979:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85979:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85980:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85980:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85981:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85983:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85985:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85987:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85988:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85989:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85990:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85885:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85867:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85865:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85874:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85883:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85884:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85892:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85909:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85919:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85921:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85922:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85926:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85943:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85947:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85959:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85974:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85976:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85978:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85979:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85980:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85945:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85945:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85945:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85945:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85992:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85992:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85992:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85992:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85880:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85880:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85880:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85880:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85949:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85949:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85949:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85949:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17227:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17227:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17227:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17227:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17275:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17275:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17275:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17275:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85944:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85944:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85944:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85944:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85893:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85893:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85893:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85893:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85895:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85895:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85895:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85895:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85890:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85890:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85890:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85890:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85986:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85986:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85986:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85986:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85866:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85866:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85866:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85866:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85875:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85875:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85875:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85875:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85923:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85923:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85923:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85923:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85924:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85924:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85924:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85924:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85984:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85984:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85984:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85984:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85941:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85941:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85941:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85941:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85938:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85938:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85938:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85938:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85942:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85942:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85942:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85942:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85916:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85916:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85916:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85916:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17266:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17266:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17266:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17266:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17311:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17311:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17311:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17311:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17263:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17263:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17263:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17263:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17220:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17220:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17220:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17220:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17254:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17254:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17254:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17254:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17305:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17305:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17305:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17305:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17302:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17302:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17302:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17302:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17222:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17222:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17222:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17222:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17272:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17272:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17272:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17272:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17273:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17273:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17273:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17273:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17226:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17226:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17226:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17226:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17288:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17288:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17288:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17288:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17224:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17224:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17224:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17224:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85977:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85977:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85977:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85977:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17289:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17289:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17289:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17289:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17320:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17320:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17320:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17320:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17344:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17344:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17344:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17344:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85956:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85956:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85956:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85956:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17223:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17223:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17223:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17223:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17303:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17303:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17303:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17303:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85967:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85967:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85967:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85967:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85889:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85889:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85889:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85889:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85940:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85940:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85940:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85940:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85925:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85925:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85925:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85925:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85920:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85920:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85920:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85920:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85935:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85935:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85935:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85935:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85877:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85877:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85877:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85877:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85907:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85907:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85907:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85907:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17299:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17299:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17299:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17299:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17318:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17318:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17318:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17318:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85975:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85975:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85975:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85975:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85936:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85936:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85936:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85936:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17329:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17329:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17329:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17329:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85982:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85982:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85982:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85982:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17316:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17316:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17316:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17316:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17309:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17309:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17309:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17309:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17334:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17334:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17334:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17334:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85946:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85946:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85946:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85946:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85930:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85930:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85930:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85930:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85888:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85888:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85888:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85888:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85897:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85897:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85897:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85897:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85963:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85963:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85963:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85963:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85903:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85903:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85903:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85903:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17342:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17342:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17342:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17342:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17217:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17217:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17217:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17217:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17250:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17250:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17250:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17250:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85881:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85881:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85881:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85881:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85886:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85886:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85886:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85886:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85912:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85912:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85912:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85912:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85913:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85913:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85913:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85913:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85991:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85991:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85991:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85991:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17248:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17248:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17248:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17248:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85908:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85908:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85908:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85908:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17219:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17219:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17219:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17219:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17298:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17298:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17298:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17298:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17282:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17282:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17282:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17282:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85928:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85928:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85928:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85928:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85961:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85961:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85961:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85961:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17251:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17251:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17251:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17251:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85902:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85902:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85902:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85902:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17280:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17280:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17280:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17280:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17253:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17253:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17253:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17253:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17281:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17281:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17281:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17281:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85948:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85948:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85948:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85948:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85873:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85873:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85873:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85873:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17227:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17275:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85910:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85910:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85910:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85910:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17237:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17237:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17237:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17237:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17220:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85870:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85870:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85870:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85870:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17263:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17327:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17327:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17327:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17327:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17311:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17266:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85904:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85904:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85904:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85904:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17331:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17331:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17331:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17331:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17301:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17301:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17301:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17301:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85901:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85901:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85901:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85901:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17325:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17325:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17325:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17325:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17254:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17265:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17265:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17265:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17265:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17305:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17302:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85945:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85992:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85880:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85966:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85966:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85966:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85966:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17216:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17216:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17216:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17216:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17339:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17339:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17339:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17339:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17227:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85949:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17319:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17319:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17319:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17319:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85990:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85990:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85990:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85990:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17275:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17220:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17263:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17335:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17335:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17335:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17335:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17244:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17244:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17244:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17244:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17311:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17321:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17321:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17321:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17321:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17266:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17222:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17254:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17290:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17290:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17290:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17290:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17269:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17269:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17269:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17269:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85916:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17305:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85945:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85992:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85941:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17272:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17302:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85880:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85942:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17273:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17226:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85924:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85952:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85952:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85952:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85952:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17227:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17227:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17296:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17296:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17296:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17296:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17340:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17340:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17340:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17340:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85965:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85965:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85965:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85965:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85950:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85950:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85950:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85950:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85957:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85957:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85957:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85957:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17220:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17227:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17220:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85938:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85923:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17275:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17220:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17275:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17263:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17263:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17220:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17227:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17275:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85949:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17263:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17275:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17263:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17257:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17257:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17257:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17257:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17311:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17263:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17275:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17220:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17227:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17224:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17266:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17336:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17336:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17336:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17336:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17311:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17247:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17247:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17247:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17247:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17288:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17266:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17311:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17266:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17222:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17311:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17266:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17311:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17266:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85984:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17254:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17220:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17263:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17254:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17255:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17255:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17255:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17255:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17263:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17263:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17263:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17220:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17220:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17220:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85895:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17254:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17254:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17275:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17227:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17266:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17311:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17275:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17275:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17275:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17227:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17227:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17227:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17254:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17266:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17266:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17266:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17311:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17311:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17311:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17264:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17264:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17264:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17264:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85943:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85943:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85943:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85943:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17254:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17305:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17254:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17254:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17254:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85983:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85983:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85983:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85983:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85909:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85909:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85909:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85909:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17233:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17233:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17233:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17233:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17286:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17286:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17286:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17286:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17305:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17305:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17272:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17305:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17305:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85911:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85911:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85911:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85911:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85962:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85962:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85962:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85962:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85868:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85868:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85868:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85868:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17302:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85944:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17289:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17305:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17243:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17243:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17243:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17243:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17326:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17326:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17326:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17326:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17302:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17305:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17305:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17305:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17273:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17302:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85960:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85960:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85960:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85960:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17284:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17284:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17284:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17284:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17302:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85973:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85973:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85973:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85973:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17229:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17229:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17229:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17229:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17302:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85905:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85905:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85905:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85905:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17320:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17226:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17303:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85986:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17302:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85878:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85878:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85878:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85878:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17223:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85947:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85947:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85947:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85947:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17302:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17302:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17302:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17241:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17241:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17241:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17241:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17246:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17246:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17246:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17246:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17344:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17230:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17230:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17230:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17230:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85893:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85867:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85867:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85867:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85867:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85980:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85980:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85980:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85980:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17224:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85890:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17308:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17308:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17308:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17308:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17288:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85978:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85978:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85978:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85978:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85875:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85919:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85919:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85919:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85919:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17228:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17228:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17228:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17228:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17287:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17287:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17287:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17287:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17222:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17222:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17222:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17222:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85892:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85892:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85892:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85892:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17222:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85959:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85959:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85959:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85959:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85974:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85974:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85974:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85974:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17222:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17245:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17245:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17245:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17245:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17222:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17222:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17222:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85884:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85884:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85884:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85884:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85979:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85979:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85979:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85979:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85964:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85964:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85964:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85964:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17262:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17262:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17262:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17262:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85891:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85891:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85891:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85891:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85921:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85921:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85921:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85921:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85916:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85883:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85883:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85883:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85883:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85922:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85922:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85922:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85922:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17313:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17313:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17313:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17313:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85941:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17304:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17304:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17304:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17304:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17218:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17218:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17218:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17218:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17285:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17285:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17285:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17285:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85866:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85876:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85876:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85876:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85876:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85898:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85898:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85898:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85898:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17240:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17240:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17240:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17240:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17306:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17306:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17306:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17306:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17272:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17272:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17291:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17291:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17291:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17291:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17272:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85942:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85887:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85887:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85887:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85887:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85899:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85899:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85899:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85899:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17272:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85992:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85992:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85915:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85915:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85915:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85915:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17272:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85914:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85914:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85914:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85914:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85951:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85951:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85951:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85951:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85945:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85945:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17272:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17272:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17272:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17272:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85969:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85969:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85969:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85969:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85953:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85953:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85953:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85953:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85885:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85885:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85885:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85885:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85894:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85894:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85894:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85894:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85900:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85900:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85900:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85900:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85896:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85896:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85896:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85896:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85906:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85906:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85906:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85906:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17238:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17238:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17238:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17238:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85918:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85918:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85918:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85918:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85981:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85981:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85981:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85981:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85882:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85882:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85882:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85882:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85926:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85926:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85926:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85926:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85937:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85937:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85937:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85937:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85992:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85927:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85927:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85927:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85927:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85945:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85970:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85970:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85970:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85970:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17289:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85879:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85879:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85879:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85879:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85872:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85872:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85872:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85872:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85933:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85933:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85933:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85933:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85972:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85972:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85972:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85972:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17323:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17323:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17323:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17323:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85924:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85869:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85869:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85869:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85869:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85954:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85954:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85954:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85954:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17273:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17273:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85934:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85934:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85934:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85934:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85939:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85939:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85939:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85939:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85989:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85989:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85989:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85989:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85874:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85874:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85874:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85874:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85968:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85968:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85968:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85968:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85931:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85931:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85931:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85931:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85958:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85958:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85958:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85958:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85871:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85871:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85871:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85871:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85932:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85932:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85932:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85932:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85987:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85987:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85987:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85987:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17273:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85917:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85917:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85917:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85917:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85985:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85985:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85985:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85985:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17318:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85988:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85988:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85988:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85988:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17273:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85955:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85955:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85955:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85955:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85971:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85971:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85971:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85971:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85929:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85929:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85929:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85929:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85976:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85976:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85976:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85976:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17273:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17322:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17322:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17322:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17322:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17333:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17333:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17333:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17333:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17267:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17267:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17267:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17267:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85880:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85880:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17283:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17283:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17283:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17283:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17309:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17226:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17261:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17261:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17261:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17261:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17226:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17273:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17223:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17303:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17273:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17273:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17273:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17312:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17312:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17312:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17312:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17320:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17226:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17226:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85880:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17300:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17300:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17300:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17300:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17226:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17259:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17259:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17259:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17259:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17226:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17226:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17226:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17226:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17297:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17297:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17297:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17297:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17338:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17338:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17338:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17338:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17344:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85923:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17328:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17328:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17328:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17328:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85938:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17316:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17307:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17307:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17307:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17307:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17294:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17294:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17294:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17294:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17299:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85977:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17232:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17232:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17232:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17232:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17239:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17239:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17239:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17239:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17242:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17242:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17242:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17242:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17274:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17274:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17274:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17274:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17224:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17224:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85984:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17224:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17224:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17256:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17256:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17256:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17256:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17224:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17236:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17236:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17236:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17236:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17231:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17231:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17231:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17231:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17221:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17221:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17221:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17221:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17260:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17260:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17260:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17260:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17343:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17343:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17343:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17343:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17224:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17224:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17224:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17224:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17330:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17330:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17330:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17330:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17252:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17252:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17252:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17252:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17293:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17293:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17293:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17293:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17334:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17234:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17234:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17234:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17234:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17295:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17295:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17295:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17295:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17225:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17225:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17225:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17225:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17288:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17288:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85895:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85949:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85949:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17270:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17270:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17270:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17270:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17329:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17288:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17337:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17337:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17337:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17337:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17271:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17271:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17271:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17271:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17288:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17249:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17249:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17249:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17249:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17258:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17258:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17258:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17258:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17292:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17292:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17292:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17292:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17288:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17317:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17317:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17317:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17317:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17341:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17341:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17341:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17341:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:85949:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17278:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17278:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17278:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17278:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17268:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17268:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17268:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17268:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17277:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17277:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17277:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17277:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17276:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17276:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17276:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17276:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17279:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17279:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17279:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17279:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17310:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17310:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17310:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17310:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17314:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17314:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17314:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17314:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17332:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17332:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17332:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17332:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17315:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17315:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17315:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17315:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17235:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:17235:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:17235:verbs:fabric:verbs_devs_print():897 10.205.73.48 libfabric:17235:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:476 libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17288:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17288:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17288:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17288:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85956:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85944:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85986:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85893:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17217:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17342:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17250:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17280:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17219:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17248:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17282:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17298:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17251:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17289:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85890:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17281:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17289:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17289:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17318:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17289:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17289:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85875:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17223:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17309:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17289:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17320:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17289:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17289:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17289:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17223:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17303:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17223:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17223:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17320:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17303:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17223:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17320:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17303:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17320:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17303:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17320:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17253:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17303:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17344:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17223:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17316:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17223:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17223:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17223:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17299:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17344:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17320:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17344:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17320:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17320:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17320:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17344:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17303:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17303:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17303:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17303:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17344:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85967:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17334:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17344:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17344:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17344:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17344:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17329:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85940:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17237:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85916:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85916:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17327:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85925:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85916:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85920:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85889:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17331:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17301:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17217:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17342:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17280:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17219:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85941:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85941:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85865:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85865:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85865:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85865:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17282:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17251:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17250:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85941:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17298:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17248:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17281:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17325:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17318:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17265:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85866:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17309:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17318:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17318:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17318:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17318:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17309:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17309:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17309:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17309:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17318:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17318:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17318:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17318:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17309:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17253:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17309:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17309:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17309:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17299:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85942:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85942:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17316:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85935:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85942:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17299:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17316:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17316:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17299:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17316:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17299:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17316:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85924:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85924:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17299:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17334:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85936:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85982:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17316:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17316:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17316:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17316:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85975:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85907:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17334:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17299:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85924:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17334:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17299:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17299:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17299:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85946:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85977:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17334:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17254:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17254:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17275:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17275:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85877:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17220:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17220:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17329:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17334:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17266:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17302:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17302:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17329:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17227:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17227:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17263:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17263:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17266:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17305:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17305:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17311:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17311:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17329:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17237:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17329:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17329:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17334:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17334:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17334:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17334:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85923:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85923:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85923:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85963:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85938:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85938:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17327:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17329:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17329:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17329:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17329:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85938:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17301:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17222:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17222:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17219:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17342:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17331:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17219:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17280:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17282:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17342:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85930:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17219:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17342:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17282:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17217:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17342:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17219:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17251:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17280:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17282:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17250:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17342:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17219:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17217:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17282:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17280:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85984:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85984:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85888:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17217:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17280:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17250:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17251:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17282:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17217:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17280:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17250:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17251:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85895:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85895:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17217:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17342:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17250:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17251:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17342:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17342:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17342:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85984:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17219:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17250:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17251:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17219:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17219:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17219:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17248:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17282:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17298:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17280:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85895:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17282:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17282:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17282:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17217:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17280:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17280:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17280:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17217:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17217:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17217:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17248:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17298:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17248:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17250:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17298:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17250:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17250:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17250:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17248:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17251:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17298:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17281:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17251:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17251:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17251:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17248:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17325:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17265:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17298:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85956:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17319:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17253:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17339:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17281:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17248:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17222:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17254:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17266:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17275:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17311:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17220:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17227:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17248:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17248:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17248:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17253:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17263:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17281:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17298:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17302:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17305:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17298:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17298:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17298:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17281:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17253:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17253:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17281:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17253:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85897:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17335:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17321:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17237:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85944:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85944:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85991:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85903:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17281:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17281:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17281:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17281:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17253:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17327:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17253:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17253:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17253:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17237:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85928:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17272:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85944:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17272:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17237:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17305:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17305:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17222:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17222:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17227:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17227:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17254:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17254:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17263:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17263:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17266:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17266:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17275:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17275:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17302:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17302:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17311:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17311:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17220:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17220:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17244:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85986:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85986:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17327:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17237:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85961:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17327:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17237:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85908:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17327:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17327:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85902:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85881:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85893:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85893:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17272:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17227:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17266:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85986:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17272:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17272:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17301:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17222:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17254:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17263:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17220:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85893:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17311:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17331:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17237:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17302:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17237:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17237:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17237:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17305:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85912:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17275:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85913:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17327:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17266:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17327:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17327:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17327:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17227:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17331:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17301:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17273:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17273:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85886:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17272:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17331:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17301:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17254:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17331:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17263:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17301:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17222:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17227:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17227:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17227:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17254:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17254:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17254:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17263:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17263:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17263:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17266:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17266:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17266:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17302:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17302:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17302:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17302:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17220:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17220:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17220:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17220:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17222:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17222:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17222:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17226:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17226:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17311:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17311:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17311:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17311:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17301:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17331:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17290:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85890:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85890:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17305:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17275:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17275:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17275:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17305:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17305:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17305:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17275:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17273:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17339:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17265:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17325:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17302:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17319:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17331:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17263:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17272:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17272:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17272:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17272:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17273:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17273:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17226:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17254:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17301:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17222:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17266:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17301:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17301:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17301:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17331:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17331:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17331:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17227:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17311:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85890:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17275:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17226:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17226:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17220:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17269:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85875:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85875:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17265:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17305:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17265:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17325:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17265:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17325:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17273:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17265:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17272:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17325:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17226:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85948:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17325:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17265:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85875:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17265:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17265:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17265:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17325:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17325:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17325:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17325:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85967:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85873:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17224:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17224:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85940:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17224:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17224:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17224:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17266:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17266:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17266:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17266:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17302:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17302:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17302:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17302:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17254:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17254:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17254:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17254:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17254:mlx:core:mlx_fabric_open():172 libfabric:17254:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17254:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17254:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17266:mlx:core:mlx_fabric_open():172 libfabric:17266:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17266:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17266:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17302:mlx:core:mlx_fabric_open():172 libfabric:17302:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17302:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17302:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17263:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17263:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17263:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17263:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17263:mlx:core:mlx_fabric_open():172 libfabric:17263:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17263:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17263:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17224:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17222:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17222:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17222:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17222:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17311:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17311:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17311:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17311:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17222:mlx:core:mlx_fabric_open():172 libfabric:17222:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17222:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17222:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17311:mlx:core:mlx_fabric_open():172 libfabric:17311:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17311:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17311:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17220:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17220:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17220:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17220:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17220:mlx:core:mlx_fabric_open():172 libfabric:17220:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17220:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17220:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17227:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17227:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17227:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17227:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17227:mlx:core:mlx_fabric_open():172 libfabric:17227:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17227:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17227:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17288:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17288:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85925:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85889:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85920:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85910:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17288:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17275:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17275:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17275:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17273:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17275:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17275:mlx:core:mlx_fabric_open():172 libfabric:17275:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17275:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17275:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17305:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17305:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17305:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17305:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17305:mlx:core:mlx_fabric_open():172 libfabric:17305:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17305:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17305:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17273:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17273:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17273:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17288:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17288:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17272:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17272:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17272:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17226:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17272:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17272:mlx:core:mlx_fabric_open():172 libfabric:17272:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17272:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17272:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17226:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17226:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17226:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17224:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17224:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17224:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17224:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17273:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17288:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17224:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17226:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85870:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85904:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17335:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85901:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17216:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85935:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85982:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85936:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85877:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85946:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85975:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85907:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17289:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17321:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17289:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17289:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17289:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17289:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17289:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85963:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17224:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17224:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17224:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17224:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17224:mlx:core:mlx_fabric_open():172 libfabric:17224:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17224:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17224:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17226:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17273:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17288:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17288:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17288:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85866:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85866:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17226:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17226:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17226:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17226:mlx:core:mlx_fabric_open():172 libfabric:17226:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17226:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17226:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17273:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17273:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17273:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17273:mlx:core:mlx_fabric_open():172 libfabric:17273:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17273:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17273:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17288:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85977:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85977:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85866:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17288:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17320:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17320:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17223:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17223:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85977:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17303:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17303:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85888:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17289:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17289:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17289:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17289:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17223:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17320:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85930:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17223:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17223:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17320:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17320:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17344:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17344:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17303:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17244:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17288:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17288:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17288:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17288:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17288:mlx:core:mlx_fabric_open():172 libfabric:17288:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17288:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17288:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17303:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17303:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17289:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17223:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17320:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17303:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17344:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17344:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17344:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17344:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85897:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85991:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17320:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17320:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17320:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17289:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17289:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17289:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17289:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17289:mlx:core:mlx_fabric_open():172 libfabric:17289:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17289:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17289:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17303:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17303:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17303:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17303:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17320:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17223:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17223:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17223:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17223:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85956:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85956:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17344:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17344:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17344:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17344:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85928:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85903:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85961:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17320:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85956:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17303:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85908:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85902:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85881:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17290:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17320:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17320:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17320:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17320:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17320:mlx:core:mlx_fabric_open():172 libfabric:17320:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17320:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17320:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85912:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17223:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85966:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17303:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17303:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17303:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17303:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17303:mlx:core:mlx_fabric_open():172 libfabric:17303:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17303:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17303:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17269:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17339:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85913:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17319:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85886:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17335:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17339:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17223:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17223:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17223:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17223:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17223:mlx:core:mlx_fabric_open():172 libfabric:17223:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17223:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17223:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17339:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17319:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17321:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17339:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17319:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17335:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17339:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17321:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17319:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17344:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17335:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17321:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17335:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17321:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85990:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17319:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17335:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17321:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85945:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85945:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85945:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85945:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17339:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85992:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85992:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85992:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85992:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85948:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17339:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17339:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17339:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17344:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17216:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17344:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17344:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17344:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17344:mlx:core:mlx_fabric_open():172 libfabric:17344:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17344:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17318:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17344:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17318:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17319:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17319:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17319:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17319:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85873:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85880:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85880:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85880:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85880:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17321:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17296:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17321:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17321:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17321:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17244:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17309:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17309:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17318:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17336:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17340:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17257:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17318:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17318:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17247:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17244:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17244:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17318:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17244:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17244:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17309:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17335:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17335:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17335:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17335:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17309:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17309:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17318:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17318:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17318:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17318:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17318:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17318:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17318:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17318:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17318:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17318:mlx:core:mlx_fabric_open():172 libfabric:17318:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17318:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17318:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17244:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17316:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17316:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17244:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17244:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17244:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17290:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17290:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17290:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17269:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17286:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17255:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17290:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17299:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17299:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17316:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85967:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85967:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17290:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17269:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17316:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17316:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17269:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85910:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17269:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85967:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17269:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17302:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17302:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85940:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85940:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17305:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17305:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17299:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17299:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17299:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85940:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17334:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17299:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17334:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17216:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85925:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85925:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17216:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85920:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85920:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85889:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85889:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17329:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17329:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17216:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17299:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17299:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17299:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17299:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17247:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17266:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17266:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17216:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17220:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17220:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17216:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17334:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17334:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17334:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17299:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85925:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85889:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17336:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17257:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85870:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17329:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17340:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85920:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17299:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17299:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17299:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17299:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17299:mlx:core:mlx_fabric_open():172 libfabric:17299:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17299:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17299:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17329:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17329:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17316:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85904:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17290:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17269:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17290:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17290:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17290:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17269:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17269:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17269:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17254:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17254:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85949:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85949:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85949:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85949:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17216:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17296:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85901:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17216:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17216:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17216:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17316:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17316:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17316:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17309:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17316:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17316:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17309:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17233:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17309:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17309:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17309:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17286:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17288:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17288:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17311:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17311:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17255:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17219:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17219:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17342:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17342:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17326:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17275:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17275:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17263:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17263:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17272:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17272:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17334:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17280:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17282:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17282:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17342:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17217:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17217:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17280:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17222:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17222:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17342:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17342:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17219:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17342:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17219:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17219:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17219:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17250:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17250:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17251:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17251:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17334:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17334:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17334:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17334:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17334:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17344:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17344:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17217:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17280:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17282:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17342:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17342:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17342:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17217:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17217:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17219:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17342:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17219:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17219:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17219:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17334:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17334:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17334:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17334:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17334:mlx:core:mlx_fabric_open():172 libfabric:17334:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17334:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17334:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17329:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17280:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17280:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17282:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17282:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85936:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85936:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17247:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17342:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17219:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17250:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17251:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17280:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17282:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17342:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17342:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17342:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17342:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17342:mlx:core:mlx_fabric_open():172 libfabric:17342:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17342:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17342:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17226:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17226:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17250:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17250:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17251:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17251:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85936:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17329:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17316:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17329:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17329:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85935:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85935:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17316:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17316:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17316:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17316:mlx:core:mlx_fabric_open():172 libfabric:17316:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17316:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17316:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17329:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17298:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17298:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85982:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85982:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17257:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17280:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85975:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85975:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17250:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17248:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17280:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17280:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17280:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17329:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17248:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17219:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17219:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17219:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17219:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17219:mlx:core:mlx_fabric_open():172 libfabric:17219:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17219:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17219:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17282:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17282:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17282:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17282:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85935:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17336:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85946:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85946:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85982:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17223:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17223:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17340:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17282:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17336:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17329:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17329:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17329:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17329:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17329:mlx:core:mlx_fabric_open():172 libfabric:17329:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17329:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17329:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85975:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17336:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85946:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85877:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85877:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17298:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17282:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17282:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17282:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17282:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17282:mlx:core:mlx_fabric_open():172 libfabric:17282:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17282:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17282:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17296:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17336:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17248:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17298:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17298:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17336:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17217:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85907:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85907:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17248:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17248:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85877:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17286:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17248:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17336:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17336:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17336:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17336:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17247:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17247:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17286:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85907:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17251:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17247:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17281:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17281:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17286:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17247:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17286:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17286:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17309:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17257:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17253:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17253:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17247:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17257:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17247:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17247:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17247:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17340:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17233:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17257:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17281:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17286:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17340:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17286:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17286:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17286:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17296:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17281:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17281:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17298:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17340:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17257:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17296:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17281:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17340:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17296:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17217:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17217:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17217:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17217:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17253:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17296:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17280:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17217:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17253:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17253:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17327:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17237:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17327:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17237:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17253:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17227:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17227:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17248:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17248:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17248:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17248:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17251:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17251:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17251:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17251:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17331:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17301:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17301:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17331:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17250:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17250:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17250:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17250:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17309:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17309:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17309:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17309:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17309:mlx:core:mlx_fabric_open():172 libfabric:17309:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17309:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17309:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17298:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17298:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17298:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17298:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17257:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17281:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17327:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17281:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17281:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17281:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17257:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17257:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17257:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17248:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17273:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17273:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85963:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85963:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17237:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17253:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17289:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17289:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17303:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17303:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17253:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17253:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17253:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17327:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17327:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17296:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17340:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17265:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17320:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17320:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17265:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17237:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17237:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17296:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17296:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17296:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17340:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17340:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17340:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17325:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17325:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17280:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17280:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17280:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17326:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17264:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17280:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17280:mlx:core:mlx_fabric_open():172 libfabric:17280:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17280:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17280:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17255:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17331:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17301:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17255:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17331:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17331:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17301:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17301:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17255:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85963:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17255:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17325:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17265:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17255:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17265:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17265:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17298:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17325:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17325:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17250:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17251:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17265:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85930:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85930:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17281:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17237:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17325:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17265:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17327:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17265:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17265:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17265:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17298:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17298:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17298:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17298:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17298:mlx:core:mlx_fabric_open():172 libfabric:17298:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17298:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17298:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85930:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17301:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17331:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17248:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17281:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17281:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17281:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17281:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17248:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17248:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17248:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17248:mlx:core:mlx_fabric_open():172 libfabric:17248:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17248:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17248:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17281:mlx:core:mlx_fabric_open():172 libfabric:17281:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17281:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17281:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17265:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85888:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85888:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17327:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17327:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17327:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17327:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17217:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17217:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17217:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17217:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17217:mlx:core:mlx_fabric_open():172 libfabric:17301:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17301:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17301:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17217:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17217:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17217:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17301:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17331:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17331:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17331:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17331:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17331:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17224:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17224:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17265:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17265:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17265:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17265:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17265:mlx:core:mlx_fabric_open():172 libfabric:17265:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17265:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17265:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85888:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17327:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17301:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17253:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17255:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17255:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17255:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17255:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17250:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17331:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17331:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17331:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17250:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17250:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17250:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17250:mlx:core:mlx_fabric_open():172 libfabric:17250:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17250:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17250:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17331:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17331:mlx:core:mlx_fabric_open():172 libfabric:17331:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17331:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17331:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17233:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17318:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17318:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17233:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17251:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17251:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17251:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17251:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17251:mlx:core:mlx_fabric_open():172 libfabric:17251:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17251:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17251:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17284:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17327:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17327:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17327:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17253:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17327:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17327:mlx:core:mlx_fabric_open():172 libfabric:17327:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17327:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17327:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17233:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17253:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17253:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17253:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17253:mlx:core:mlx_fabric_open():172 libfabric:17253:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17253:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17253:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17233:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17233:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17325:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17325:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17325:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17325:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17237:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17237:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17237:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17237:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17237:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17233:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17233:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17233:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17233:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17264:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17237:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17237:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17237:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17237:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17237:mlx:core:mlx_fabric_open():172 libfabric:17237:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17237:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17237:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17229:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17243:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17301:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17301:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17301:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85897:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85897:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17301:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17301:mlx:core:mlx_fabric_open():172 libfabric:17301:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17301:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17301:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17326:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17241:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85897:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17326:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17246:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17326:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17326:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17325:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17326:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85991:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85991:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17325:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17325:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17325:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17325:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17325:mlx:core:mlx_fabric_open():172 libfabric:17325:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17325:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17325:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17326:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85991:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17326:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17326:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17326:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17284:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85966:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85961:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85961:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85928:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85928:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85961:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85928:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85903:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85903:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85902:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85902:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17264:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85908:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85908:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17264:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17229:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85903:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17264:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85902:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17282:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17282:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17264:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85908:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85916:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85916:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85916:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85916:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17230:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17264:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17342:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17342:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17264:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17287:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17264:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17264:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17264:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85941:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85941:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85941:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85941:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85881:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85881:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17246:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85912:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85912:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17243:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17316:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17316:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85990:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17302:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85881:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85912:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17241:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17301:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17301:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17334:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17334:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85913:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85913:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17265:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17265:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85886:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85886:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85913:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85886:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85952:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17298:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17298:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17250:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17250:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17281:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17281:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17331:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17331:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17329:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17329:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17219:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17219:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85965:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17299:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17299:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17230:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17248:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17248:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17284:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17309:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17309:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17284:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17284:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17287:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17284:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17284:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17284:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17327:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17327:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17284:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17284:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17284:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85948:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85948:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17325:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17325:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85948:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17254:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85873:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85873:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17243:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17272:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17246:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85873:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17305:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17306:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17308:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85942:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85942:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85942:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85942:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17246:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17246:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17280:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17280:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17246:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85950:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17246:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17229:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17229:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85957:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17229:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17237:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17237:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17246:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17229:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17243:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17246:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17246:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17246:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17229:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17243:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17243:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17243:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17344:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17229:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17253:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17253:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17229:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17229:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17229:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17243:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17243:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17243:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17243:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17217:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17217:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17339:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17339:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17319:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17319:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17321:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17321:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85924:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85924:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85924:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85924:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17251:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17251:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17339:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17339:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17339:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17304:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17285:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17319:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17321:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17288:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17226:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17319:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17319:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17321:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17321:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17339:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17320:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17335:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17335:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17319:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17321:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17303:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17230:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17335:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17339:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17339:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17339:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17339:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17230:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17335:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17335:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17230:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17230:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17339:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17230:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17287:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17311:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17244:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17244:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17319:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17319:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17319:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17319:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17335:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17321:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17321:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17321:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17321:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17287:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17230:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17287:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17230:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17230:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17230:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17319:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85923:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85923:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85923:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85923:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17241:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17312:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17287:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17321:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17287:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17241:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17241:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17241:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17244:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17287:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17241:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17287:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17287:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17287:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17244:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17244:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17244:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17241:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85910:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85910:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17241:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17241:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17241:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17339:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17339:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17339:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17339:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17339:mlx:core:mlx_fabric_open():172 libfabric:17339:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17339:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17339:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17262:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85910:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85909:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17335:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17335:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17335:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17335:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17335:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17290:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17290:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85938:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85938:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85938:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85938:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17319:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17319:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17319:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17319:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17319:mlx:core:mlx_fabric_open():172 libfabric:17319:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17319:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17319:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17321:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17321:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17321:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17321:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17321:mlx:core:mlx_fabric_open():172 libfabric:17321:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17321:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17321:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17269:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17269:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17290:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17290:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17290:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17290:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85911:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17216:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17216:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17269:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17228:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17269:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17269:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17269:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17244:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17244:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17244:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17244:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17244:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17216:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17216:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17216:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17216:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17335:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17335:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17335:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17266:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17335:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17335:mlx:core:mlx_fabric_open():172 libfabric:17335:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17335:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17335:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17275:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85904:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85904:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17220:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85984:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85984:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85984:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85984:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85904:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17263:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85895:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85895:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85895:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85895:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85943:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85870:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85870:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17290:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17290:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17290:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17290:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17306:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17290:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17308:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85905:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85878:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17222:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85870:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85973:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85901:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85901:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17330:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85901:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17269:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17269:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17269:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17269:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17269:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17289:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85983:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17244:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17244:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17244:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85960:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17244:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17244:mlx:core:mlx_fabric_open():172 libfabric:17244:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17244:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17244:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17336:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17336:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85962:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17216:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17216:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17216:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17216:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17216:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17247:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17247:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17336:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17286:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17286:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17336:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17336:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17336:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17227:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17247:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17247:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17247:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17286:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17247:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17286:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17286:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17286:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17290:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17290:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17290:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17290:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17290:mlx:core:mlx_fabric_open():172 libfabric:17290:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17290:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17290:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85944:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85944:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85944:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85944:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85947:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85986:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85986:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85986:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85986:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85868:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85893:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85893:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85893:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85893:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17269:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17269:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17269:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17269:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17269:mlx:core:mlx_fabric_open():172 libfabric:17269:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17269:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17269:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17257:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17257:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17340:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17340:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17296:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17296:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85980:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17216:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17216:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17216:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17216:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17216:mlx:core:mlx_fabric_open():172 libfabric:17216:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17216:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17216:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17257:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85919:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17340:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17257:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17257:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17257:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17296:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85890:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85890:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85890:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85890:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17340:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17340:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17340:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17296:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17296:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17296:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85978:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17336:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17336:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17336:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17336:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17336:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85875:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85875:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85875:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85875:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85892:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17255:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17255:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17223:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17247:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17247:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17247:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17247:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17247:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17286:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17286:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17286:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17286:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17286:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17255:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17224:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17255:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17255:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17255:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17343:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85884:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17233:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17233:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85867:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17333:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17323:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17233:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17233:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17233:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17233:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85891:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85974:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85883:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85959:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17304:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17257:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17257:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17257:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17257:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17257:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17296:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17296:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17296:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17296:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17296:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17340:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17340:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17340:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17340:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17340:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17301:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17273:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17326:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17326:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17336:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17336:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17336:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17336:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17336:mlx:core:mlx_fabric_open():172 libfabric:17336:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17336:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17336:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85921:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17285:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85922:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17247:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17247:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17247:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17326:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17247:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17247:mlx:core:mlx_fabric_open():172 libfabric:17247:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17247:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17247:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17326:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17326:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17326:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17293:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17286:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17286:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17286:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17286:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17286:mlx:core:mlx_fabric_open():172 libfabric:17286:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17286:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17286:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85979:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85964:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17255:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17255:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17255:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17255:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17255:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17298:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17264:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17264:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17282:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17233:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17233:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17233:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17233:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17233:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17264:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17264:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17264:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17264:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85876:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85887:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85898:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17257:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17257:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17257:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17257:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85915:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17257:mlx:core:mlx_fabric_open():172 libfabric:17257:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17257:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17257:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17340:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17340:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17340:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17340:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17340:mlx:core:mlx_fabric_open():172 libfabric:17340:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17340:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17340:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17296:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17296:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17296:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17296:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17296:mlx:core:mlx_fabric_open():172 libfabric:17296:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17296:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17296:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17265:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17318:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17316:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17326:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17326:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17326:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17326:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17326:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85914:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85885:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85866:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85866:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85866:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85866:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17312:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17255:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17255:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17255:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85977:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85977:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85977:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85977:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17255:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17255:mlx:core:mlx_fabric_open():172 libfabric:17255:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17255:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17255:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17334:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85872:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17329:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85894:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17331:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85896:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85953:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85926:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85906:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85969:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85918:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17339:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17339:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85951:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85900:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85879:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85869:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85956:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85956:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85956:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85956:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85927:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85972:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17233:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17233:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17233:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17233:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17233:mlx:core:mlx_fabric_open():172 libfabric:17233:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17233:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17233:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17264:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17264:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17264:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17264:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17264:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85937:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85970:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17284:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17284:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85933:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85899:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85917:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17319:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17319:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85882:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85958:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17284:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17284:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17284:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85981:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17284:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17245:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85932:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85955:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85871:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85939:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85971:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85934:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17262:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17321:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17321:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17281:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85989:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85976:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85966:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85966:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17326:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17326:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17326:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17326:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17342:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17326:mlx:core:mlx_fabric_open():172 libfabric:17326:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17326:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17326:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17291:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85966:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17219:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17248:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17250:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17299:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17325:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85874:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85931:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17309:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17228:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17229:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17229:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17246:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17246:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17243:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17243:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17264:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17280:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17264:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17264:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17264:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17264:mlx:core:mlx_fabric_open():172 libfabric:17264:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17264:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17264:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85968:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17327:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17237:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17297:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17246:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17335:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17335:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17229:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17310:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17243:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17246:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17246:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17246:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17284:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17284:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17284:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17284:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17229:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17229:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17284:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17229:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17243:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17243:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17243:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17337:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17256:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17294:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17322:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85929:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17217:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17253:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85954:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85952:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17308:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17308:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85987:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17306:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17306:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17308:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17283:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17308:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17306:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17251:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17306:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17308:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17306:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85967:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85967:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85967:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85967:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17308:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85988:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85985:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17308:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17308:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17308:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17330:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17306:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17267:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17306:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17306:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17306:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85965:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17300:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17244:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17244:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17292:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17284:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17284:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17284:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17295:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17284:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17284:mlx:core:mlx_fabric_open():172 libfabric:17284:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17284:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17284:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17243:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17243:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17243:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17243:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17243:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17246:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17246:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17246:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17246:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17246:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17229:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17229:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17229:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17229:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17229:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17261:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17313:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85990:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85990:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17218:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17230:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17230:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17240:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17249:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85990:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17290:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17290:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17287:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17287:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17338:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17307:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17259:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17328:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17269:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17269:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17230:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17274:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17241:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17241:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17230:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17230:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17230:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17260:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17317:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17332:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17271:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17287:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17238:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17287:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17287:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17287:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17241:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17241:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17241:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17241:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17216:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17216:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85940:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85940:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85940:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85940:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85950:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17243:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17243:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17243:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17243:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17243:mlx:core:mlx_fabric_open():172 libfabric:17243:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17243:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17243:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17229:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17229:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17229:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17229:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17229:mlx:core:mlx_fabric_open():172 libfabric:17229:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17229:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17229:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17246:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17246:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17246:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17246:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17246:mlx:core:mlx_fabric_open():172 libfabric:17246:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17246:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17246:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17231:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17341:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85957:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17221:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17230:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17230:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17230:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17230:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17230:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85925:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85925:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85925:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85925:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17287:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17287:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17287:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17287:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17287:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85889:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85889:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85889:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85889:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17242:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17343:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17315:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17241:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17268:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17241:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17241:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17241:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17241:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85920:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85920:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85920:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85920:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17225:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17333:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17323:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17276:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17304:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17304:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17304:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17247:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17247:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17336:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17336:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17286:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17286:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17304:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17314:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17239:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17304:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17232:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17304:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17304:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17304:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17304:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17230:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17230:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17230:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17230:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17230:mlx:core:mlx_fabric_open():172 libfabric:17230:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17230:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17230:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17252:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17287:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17287:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17287:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17287:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17287:mlx:core:mlx_fabric_open():172 libfabric:17287:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17287:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17287:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17285:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17285:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17285:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17293:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17270:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17285:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17285:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17241:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17241:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17241:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17241:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17241:mlx:core:mlx_fabric_open():172 libfabric:17241:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17241:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17241:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17285:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17285:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17285:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17285:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17296:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17296:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85936:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85936:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85936:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85936:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17279:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17257:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17257:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17340:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17340:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17234:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17235:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17236:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85909:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17255:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17255:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85911:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17278:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85907:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85907:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85907:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85907:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85946:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85946:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85946:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85946:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85975:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85975:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85975:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85975:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85877:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85877:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85877:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85877:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17312:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17233:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17233:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17312:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85982:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85982:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85982:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85982:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17312:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85935:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85935:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85935:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85935:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17312:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17312:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17312:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17312:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17312:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17312:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17258:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17326:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17326:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85943:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85905:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85973:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85878:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85962:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17245:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85960:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17264:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17264:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17262:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17262:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85983:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85963:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85963:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85963:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85963:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17262:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17262:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17291:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17262:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17262:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17262:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17262:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17262:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17228:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17228:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17228:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17277:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17228:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85930:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85930:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85930:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85930:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17228:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17228:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17228:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17228:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17228:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17297:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17256:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85888:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85888:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85888:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85888:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17294:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17310:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85947:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17337:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17322:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17284:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17284:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17330:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17330:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17330:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17330:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17283:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17330:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85919:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17330:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17295:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17300:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17267:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17330:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17330:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17330:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17240:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85980:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17292:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17218:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17261:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17313:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17259:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17249:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17274:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17307:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17339:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17328:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17338:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17246:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17246:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17260:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17317:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17243:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17243:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85978:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17229:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17229:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17238:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85892:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17271:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17332:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85884:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17231:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17319:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17321:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85908:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85908:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85908:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85908:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17341:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85897:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85897:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85897:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85897:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85902:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85902:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85902:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85902:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85928:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85928:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85928:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85928:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85991:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85991:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85991:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85991:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85883:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85903:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85903:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85903:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85903:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85891:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85961:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85961:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85961:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85961:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85886:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85886:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85886:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85886:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85959:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85974:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85881:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85881:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85881:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85881:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85912:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85912:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85912:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85912:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17221:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85913:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85913:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85913:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85913:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85921:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17343:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17343:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17242:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17343:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17343:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17343:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17315:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17268:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17343:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85922:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85964:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17343:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17343:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17343:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85868:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17230:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17230:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17335:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17287:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17287:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85948:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85948:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85948:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85948:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85867:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85979:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17333:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17333:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17323:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17323:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17314:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17276:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17333:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85873:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85873:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85873:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85873:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17323:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17225:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17333:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17323:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17252:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17323:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17333:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17239:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17241:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17241:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17333:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17323:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17333:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17333:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17333:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17323:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17323:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17323:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17244:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17232:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17290:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85887:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17235:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85876:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17269:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17293:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17293:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85898:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17293:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17279:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17293:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85915:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17293:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17270:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17234:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17236:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17293:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17293:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17293:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17293:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17308:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17308:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85872:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85953:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17306:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17306:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17278:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85885:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85914:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85896:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85894:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85969:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85906:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17308:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17216:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85926:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17308:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17308:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17306:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17308:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85918:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85951:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17306:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17306:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17306:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85900:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85972:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85927:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85910:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85910:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85910:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85910:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85879:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85970:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85917:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85882:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85899:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85933:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85937:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17258:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85955:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85958:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85981:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85869:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85871:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85932:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85939:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85971:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85934:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85976:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85904:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85904:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85904:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85904:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17245:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17245:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17245:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85989:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17245:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17308:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17308:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17308:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17308:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85931:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17308:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17245:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85968:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85874:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17306:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17306:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17306:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17291:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17306:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17306:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17291:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17247:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17245:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17245:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17245:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17245:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17291:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85901:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85901:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85901:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85901:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85870:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85870:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85870:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85870:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17336:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17291:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17277:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17291:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17286:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85929:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17291:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85954:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17291:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17291:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17291:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85988:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17297:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17297:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17297:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17297:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17256:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17256:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17294:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17294:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17297:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17308:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17308:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17308:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17308:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17308:mlx:core:mlx_fabric_open():172 libfabric:17308:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17308:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17308:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17256:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17310:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17294:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17310:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17337:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17337:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17256:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17294:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17306:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17306:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17306:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17306:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17306:mlx:core:mlx_fabric_open():172 libfabric:17306:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17306:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17306:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17310:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85985:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85987:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17337:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17256:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17294:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17297:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17296:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17310:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17322:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17322:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17337:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17257:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17297:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17297:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17297:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17340:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17310:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17322:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17337:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17322:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17256:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17294:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17256:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17256:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17256:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17294:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17294:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17294:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17310:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17322:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17255:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17240:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17283:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17283:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17310:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17310:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17310:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17218:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17218:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17240:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17337:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17292:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17240:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17292:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17283:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17295:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17300:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17300:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17322:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17295:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17337:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17337:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17337:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17218:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17267:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17267:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17292:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17322:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17322:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17322:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17233:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17240:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17283:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17295:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17259:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17259:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17218:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17249:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17249:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17274:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17274:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17300:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17267:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17292:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17261:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17261:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17295:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17240:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17283:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17313:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17313:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17307:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17307:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17317:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17317:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17328:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17328:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17218:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17259:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17300:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17267:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17292:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17249:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17274:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17261:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17295:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17338:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17338:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17260:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17260:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17307:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17313:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17328:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17238:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17259:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17317:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17238:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17300:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17249:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17261:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17267:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17274:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17338:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17271:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17307:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17313:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17326:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17240:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17260:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17271:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17317:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17328:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17238:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17259:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17332:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17332:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17261:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17274:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17338:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17240:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17240:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17240:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17249:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17313:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17260:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17307:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17271:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17238:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17283:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17317:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17328:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17332:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17304:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17304:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17338:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17271:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17341:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17341:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17231:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17231:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17260:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17218:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17283:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17283:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17283:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17332:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17238:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17341:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17218:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17218:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17218:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17271:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17292:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17231:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17295:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17264:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17292:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17292:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17292:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17332:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17295:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17295:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17295:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17341:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17231:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17300:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17341:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17231:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17221:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17221:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17242:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17242:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17315:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17315:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17300:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17300:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17300:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17304:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17242:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17221:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17225:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17225:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17315:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17304:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17304:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17242:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17268:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17268:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17304:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17252:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17252:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17267:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17314:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17314:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17221:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17315:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17276:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17276:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17225:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17259:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17242:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17267:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17267:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17267:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17268:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17252:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17315:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17225:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17239:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17239:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17259:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17259:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17259:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17314:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17221:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17268:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17276:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17284:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17304:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17304:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17304:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17304:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17304:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17252:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17314:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17232:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17232:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17274:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17276:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17225:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17239:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17243:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17261:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17268:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17235:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17235:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17274:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17274:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17274:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17313:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17252:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17261:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17261:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17261:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17307:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17239:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17270:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17304:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17304:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17304:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17314:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17229:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17246:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17270:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17276:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17279:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17279:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17304:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17304:mlx:core:mlx_fabric_open():172 libfabric:17304:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17304:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17304:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17313:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17313:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17313:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17232:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17236:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17236:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17307:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17307:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17307:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17235:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17285:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17234:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17234:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17249:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17285:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17328:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17239:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17270:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17317:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17279:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17338:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17232:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17235:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17249:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17249:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17249:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17260:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17260:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17260:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17260:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17236:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17238:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17238:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17238:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17238:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17271:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17271:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17271:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17271:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17278:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17278:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17317:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17317:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17317:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17328:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17328:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17328:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17338:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17338:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17338:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17234:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17332:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17332:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17332:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17332:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17258:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17258:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17270:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17341:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17279:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17341:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17341:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17341:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17232:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17235:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17236:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17277:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17277:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17278:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17231:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17234:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17258:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17231:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17231:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17231:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17270:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17279:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85952:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85952:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17236:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17278:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17230:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17277:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17242:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17287:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17258:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17315:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17221:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17234:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17242:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17242:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17242:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17278:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17277:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17285:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17315:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17315:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17315:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17221:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17221:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17221:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17225:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17268:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17258:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17225:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17225:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17225:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17252:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17268:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17268:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17268:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17314:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17276:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17314:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17314:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17314:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17252:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17252:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17252:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17277:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17285:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17285:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17276:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17276:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17276:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17285:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85952:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17285:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17239:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17285:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17285:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17285:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17285:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17239:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17239:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17239:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17308:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17308:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17232:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17241:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17306:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17306:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17232:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17232:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17232:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17235:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17270:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17285:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17235:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17235:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17235:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17279:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17285:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17285:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17285:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17285:mlx:core:mlx_fabric_open():172 libfabric:17285:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17285:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17285:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17236:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17270:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17270:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17270:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17278:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17279:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17279:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17279:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17234:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17236:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17236:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17236:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17278:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17278:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17278:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17234:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17234:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17234:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17258:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:17277:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:17258:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17258:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17258:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17277:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:17277:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:17277:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85965:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85965:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85965:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17304:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17304:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85950:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85950:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85950:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17312:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17312:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17285:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17285:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17312:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17312:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17312:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17312:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85957:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85957:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17312:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17312:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17312:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17312:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17312:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17312:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17312:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17312:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17312:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17312:mlx:core:mlx_fabric_open():172 libfabric:17312:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17312:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17312:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85957:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17262:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17262:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17306:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17308:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17312:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17312:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17262:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17262:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17262:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17262:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17228:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17228:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17262:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17262:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17262:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17262:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17262:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17262:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17262:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17262:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17262:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17262:mlx:core:mlx_fabric_open():172 libfabric:17262:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17262:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17262:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17304:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17228:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17228:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17228:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17228:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17228:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17228:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17228:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17228:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17228:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17228:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17228:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17228:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17228:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17228:mlx:core:mlx_fabric_open():172 libfabric:17228:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17228:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17228:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85909:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85909:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17285:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85909:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17330:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17330:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17262:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17262:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17330:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17330:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17330:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17330:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17330:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17330:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17330:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17330:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17330:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85911:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85911:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17330:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17330:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17330:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17330:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17330:mlx:core:mlx_fabric_open():172 libfabric:17330:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17330:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17330:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85911:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17228:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17228:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17312:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85943:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85943:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85878:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85878:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17330:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17330:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85962:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85962:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85905:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85905:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85943:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85878:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85962:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85905:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85983:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85983:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85973:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85973:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85960:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85960:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17343:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17343:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85983:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85973:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85960:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17343:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17343:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17343:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17343:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17343:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17343:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17343:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17343:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17343:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17343:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17343:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17343:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17343:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17343:mlx:core:mlx_fabric_open():172 libfabric:17343:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17343:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17343:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17333:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17333:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17323:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17323:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85966:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85966:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85966:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85966:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17262:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85947:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85947:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17333:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17323:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17323:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17323:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17323:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17333:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17333:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17333:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85947:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17323:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17323:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17323:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17323:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17333:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17333:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17333:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17333:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17323:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17333:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17323:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17323:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17323:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17323:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17323:mlx:core:mlx_fabric_open():172 libfabric:17323:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17323:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17323:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17333:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17333:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17333:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17333:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17333:mlx:core:mlx_fabric_open():172 libfabric:17333:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17333:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17333:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17228:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85919:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85919:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85980:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85980:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85919:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17293:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17293:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17343:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17343:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85978:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85978:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85980:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17293:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85978:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17293:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17293:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17293:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85892:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85892:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17293:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17293:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17293:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17293:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17293:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17293:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17293:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17293:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17293:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17293:mlx:core:mlx_fabric_open():172 libfabric:17293:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17293:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17293:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85892:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85990:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85990:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85990:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85990:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:17330:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85884:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85884:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17323:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17323:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17333:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17333:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85884:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85883:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85883:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85891:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85891:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85891:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85883:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85974:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85974:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85959:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85959:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85959:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85974:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17293:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17293:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85921:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85921:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85964:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85964:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85921:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85964:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85979:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85979:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85922:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85922:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85979:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85868:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85868:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85922:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85868:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85887:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85887:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17245:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17245:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85876:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85876:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85898:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85898:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85867:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85867:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85887:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17245:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17291:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17291:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17245:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17245:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17245:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85876:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85915:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85915:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85867:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17245:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17245:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17245:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17245:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85898:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17245:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85872:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85872:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85953:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85953:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85969:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85969:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17245:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17245:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17245:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17245:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85992:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17245:mlx:core:mlx_fabric_open():172 libfabric:17245:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17245:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17245:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85885:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85885:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85914:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85914:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85896:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85896:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85894:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85894:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85926:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85926:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17291:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85945:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85906:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85906:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17291:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17291:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17291:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85915:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85972:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85972:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85951:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85951:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85918:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85918:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85927:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85927:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85970:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85970:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17343:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17291:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85900:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85900:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17291:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17291:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17291:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17291:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85917:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85917:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85981:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85981:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85882:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85882:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17297:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17297:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85899:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85899:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17291:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17291:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17291:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17291:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17291:mlx:core:mlx_fabric_open():172 libfabric:17291:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17291:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17291:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85933:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85933:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85937:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85937:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85958:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85958:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85955:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85955:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85869:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85869:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85871:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85871:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85971:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85971:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85932:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85932:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85976:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85976:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85939:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85939:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17294:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17294:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85934:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85934:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17256:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17256:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17333:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85872:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85931:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85931:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17310:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17310:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85874:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85874:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85968:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85968:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85989:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85989:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85929:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85929:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85953:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17297:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17337:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17337:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85987:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85987:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85988:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85988:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85954:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85954:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17297:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17297:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85969:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17297:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17322:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17322:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85880:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85885:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85914:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85985:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85985:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85879:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85879:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17294:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17297:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17297:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17297:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17297:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17256:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17297:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17294:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17294:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17294:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17310:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17256:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17256:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17256:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17297:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17297:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17297:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17297:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17297:mlx:core:mlx_fabric_open():172 libfabric:17297:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17297:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17297:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17310:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17310:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17310:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17294:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17256:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17294:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17294:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17294:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17294:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17337:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17256:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17256:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17256:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17322:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17240:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17256:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17240:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17310:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17323:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17310:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17310:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17310:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17310:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17322:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17322:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17337:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17337:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17337:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17294:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17294:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17294:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17294:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17322:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17256:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17256:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17256:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17256:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17294:mlx:core:mlx_fabric_open():172 libfabric:17294:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17294:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17294:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17256:mlx:core:mlx_fabric_open():172 libfabric:17256:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17256:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17256:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17283:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17310:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17310:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17283:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17310:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17310:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17310:mlx:core:mlx_fabric_open():172 libfabric:17310:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17310:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17310:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17337:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17322:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17322:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17322:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17337:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17337:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17337:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17337:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17322:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17322:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17292:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17218:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17218:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17292:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17295:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17295:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17337:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17322:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17322:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17322:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17322:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17337:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17337:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17337:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17337:mlx:core:mlx_fabric_open():172 libfabric:17337:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17337:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17337:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17322:mlx:core:mlx_fabric_open():172 libfabric:17322:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17322:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17322:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17240:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17291:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17291:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17300:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17300:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17245:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17245:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17240:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17240:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17240:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17293:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17283:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85865:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17283:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17283:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85894:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85896:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17283:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85906:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85917:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85918:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85926:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17292:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85949:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85951:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85970:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85972:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85927:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:17240:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17295:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17240:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17240:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17240:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17292:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17292:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17240:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17292:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17283:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17283:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17283:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17218:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17283:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17283:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17295:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17295:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17295:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17259:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17259:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17267:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17267:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17218:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17218:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17218:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17240:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17240:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17240:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17240:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17240:mlx:core:mlx_fabric_open():172 libfabric:17274:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17274:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17300:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17240:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17240:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17240:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17283:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17283:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17283:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17283:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17283:mlx:core:mlx_fabric_open():172 libfabric:17283:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17283:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17283:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17292:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17292:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17292:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17292:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17292:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17295:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17295:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17295:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17295:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17295:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17300:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17300:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17300:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17218:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17218:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17218:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17218:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17218:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17292:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17292:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17292:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17292:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17292:mlx:core:mlx_fabric_open():172 libfabric:17292:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17292:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17292:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17261:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17261:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17295:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17295:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17295:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17295:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17295:mlx:core:mlx_fabric_open():172 libfabric:17295:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17295:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17295:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17313:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17313:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17218:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17218:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17218:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17218:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17218:mlx:core:mlx_fabric_open():172 libfabric:17218:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17218:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17218:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17300:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17300:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17300:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17300:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17300:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17307:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17307:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17300:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17300:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17300:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17300:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17300:mlx:core:mlx_fabric_open():172 libfabric:17300:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17300:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17300:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17259:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17267:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17274:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17259:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17259:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17259:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17267:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17267:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17267:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17274:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17274:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17274:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17261:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17259:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17267:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17274:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17274:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17274:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17259:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17259:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17259:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17267:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17267:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17267:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17274:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17313:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17259:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17261:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17261:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17267:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17274:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17261:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17307:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17313:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17313:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17313:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17307:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17307:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17259:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17259:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17267:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17267:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17267:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17267:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17267:mlx:core:mlx_fabric_open():172 libfabric:17267:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17267:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17267:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17274:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17274:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17274:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17274:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17307:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17242:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17242:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17252:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17252:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17259:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17259:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17259:mlx:core:mlx_fabric_open():172 libfabric:17259:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17259:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17259:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17261:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17261:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17261:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17261:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17261:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17268:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17268:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17271:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17271:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17274:mlx:core:mlx_fabric_open():172 libfabric:17274:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17274:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17274:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85869:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17221:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17221:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85871:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17225:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17225:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85874:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17249:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17249:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85879:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17260:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17260:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85882:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17313:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17313:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17313:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17313:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17313:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85899:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17314:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17314:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85900:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17328:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17328:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85929:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17338:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17338:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85931:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17341:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17341:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85932:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17231:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17231:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85933:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17315:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17315:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85934:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17317:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17317:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85937:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17332:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17332:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85939:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85945:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85954:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17238:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17238:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85955:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:17276:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17276:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85958:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85968:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85971:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85976:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85981:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85985:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85987:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85988:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85989:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85992:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17307:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17307:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17307:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17307:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17307:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17261:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17261:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17261:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17261:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17261:mlx:core:mlx_fabric_open():172 libfabric:17261:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17261:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17261:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17313:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17313:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17313:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17313:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85916:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17313:mlx:core:mlx_fabric_open():172 libfabric:17313:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17313:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17313:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17239:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17239:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17235:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17235:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17278:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17278:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17307:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17307:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17307:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17307:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17307:mlx:core:mlx_fabric_open():172 libfabric:17307:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17307:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17307:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85880:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17236:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17236:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17270:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17270:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85941:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17279:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17279:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17242:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17252:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17221:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17268:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17271:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17322:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17322:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17256:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17256:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17294:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17294:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17297:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17297:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17314:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17315:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17249:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17338:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17252:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17252:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17252:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17242:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17242:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17242:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17260:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17271:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17271:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17271:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17310:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17310:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17225:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17268:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17268:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17268:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17317:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17328:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17341:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17221:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17221:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17221:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17231:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17239:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17270:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17278:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17314:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17314:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17314:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17315:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17315:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17315:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17238:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17249:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17249:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17249:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17279:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17332:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17235:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17260:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17260:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17276:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17338:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17338:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17338:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17225:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17225:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17234:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17234:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17260:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85949:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17225:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17283:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17283:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17292:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17292:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17295:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17295:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17317:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17317:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17317:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17337:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17337:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17236:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17239:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17239:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17252:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17252:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17252:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17277:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17277:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17328:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17328:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17328:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17341:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17341:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17341:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17231:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17231:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17231:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17239:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17242:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17242:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17242:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17252:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17252:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17270:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17270:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17270:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17271:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17271:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17271:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17242:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17242:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17271:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17271:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17278:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17278:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17278:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17218:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17218:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17268:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17268:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17268:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17332:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17332:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17332:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17221:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17235:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17235:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17235:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17268:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17268:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17279:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17279:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17279:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17314:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17314:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17314:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17221:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17221:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17221:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17221:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17238:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17238:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17238:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17276:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17276:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17276:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17314:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17314:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17315:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17315:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17315:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17315:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17315:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17236:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17236:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17236:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17249:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17249:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17249:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17249:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17249:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17260:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17260:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17260:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17338:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17260:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17260:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17338:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17338:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17338:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17338:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17317:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17225:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17317:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17317:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17317:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17225:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17225:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17225:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17317:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17328:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17328:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17328:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17225:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17234:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17328:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17239:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17328:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17341:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17341:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17341:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17277:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17341:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17341:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17231:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17239:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17239:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17239:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17231:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17231:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17231:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17239:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17242:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17242:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17242:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17242:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17270:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17231:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17234:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17234:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17242:mlx:core:mlx_fabric_open():172 libfabric:17242:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17242:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17242:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17278:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17234:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17270:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17270:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17252:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17252:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17270:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17278:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17278:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17278:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17252:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17252:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17252:mlx:core:mlx_fabric_open():172 libfabric:17252:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17252:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17252:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17270:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17277:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17277:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17277:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17278:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17271:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17271:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17271:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17271:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17268:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17271:mlx:core:mlx_fabric_open():172 libfabric:17271:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17271:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17271:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17268:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17268:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17268:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17268:mlx:core:mlx_fabric_open():172 libfabric:17268:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17268:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17268:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17332:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17332:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17332:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17332:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17235:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17332:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17238:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17279:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17235:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17235:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17235:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17276:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17221:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17238:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17238:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17238:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17279:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17279:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17279:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17221:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17221:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17221:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17221:mlx:core:mlx_fabric_open():172 libfabric:17235:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17276:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17276:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17276:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17276:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17221:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17221:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17221:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17238:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17279:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17236:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17236:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17236:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17314:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17236:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17314:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17314:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17236:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17314:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17314:mlx:core:mlx_fabric_open():172 libfabric:17314:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17314:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17314:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17315:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17315:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17315:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17315:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17315:mlx:core:mlx_fabric_open():172 libfabric:17315:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17315:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17315:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17249:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17249:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17249:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17249:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17260:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17249:mlx:core:mlx_fabric_open():172 libfabric:17249:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17249:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17249:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17260:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17260:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17260:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17260:mlx:core:mlx_fabric_open():172 libfabric:17260:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17260:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17260:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17338:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17338:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17338:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17338:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17338:mlx:core:mlx_fabric_open():172 libfabric:17338:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17338:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17338:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17317:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17317:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17317:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17317:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17317:mlx:core:mlx_fabric_open():172 libfabric:17317:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17317:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17225:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17232:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17317:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17225:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17225:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17225:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17232:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17328:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17328:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17225:mlx:core:mlx_fabric_open():172 libfabric:17225:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17225:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17225:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17328:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17328:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17328:mlx:core:mlx_fabric_open():172 libfabric:17328:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17341:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17341:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17341:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17328:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17328:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17341:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17341:mlx:core:mlx_fabric_open():172 libfabric:17341:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17341:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17341:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17239:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17234:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17239:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17239:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17239:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17231:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17231:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17239:mlx:core:mlx_fabric_open():172 libfabric:17239:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17239:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17239:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17278:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17231:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17231:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17231:mlx:core:mlx_fabric_open():172 libfabric:17231:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17234:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17234:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17234:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17277:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17278:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17278:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17278:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17231:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17231:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17234:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17270:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17277:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17277:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17277:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17278:mlx:core:mlx_fabric_open():172 libfabric:17278:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17278:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17278:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17332:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17332:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17332:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17332:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17235:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17270:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17270:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17270:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17279:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17279:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17279:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17332:mlx:core:mlx_fabric_open():172 libfabric:17332:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17332:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17332:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17235:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17235:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17235:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17235:mlx:core:mlx_fabric_open():172 libfabric:17236:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17236:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17236:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17238:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17238:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17238:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17238:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17270:mlx:core:mlx_fabric_open():172 libfabric:17270:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17270:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17270:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17276:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17276:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17276:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17277:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17279:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17279:mlx:core:mlx_fabric_open():172 libfabric:17279:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17279:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17279:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17235:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17235:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17235:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17236:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17236:mlx:core:mlx_fabric_open():172 libfabric:17236:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17236:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17236:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17238:mlx:core:mlx_fabric_open():172 libfabric:17238:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17238:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17238:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17276:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17276:mlx:core:mlx_fabric_open():172 libfabric:17276:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17276:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17276:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17258:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:17258:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:17234:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17234:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17234:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17234:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17234:mlx:core:mlx_fabric_open():172 libfabric:17234:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17234:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17234:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17277:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17277:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17277:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17277:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17277:mlx:core:mlx_fabric_open():172 libfabric:17277:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17277:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17277:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17232:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17258:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:17232:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17232:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17232:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17258:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:17258:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17258:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17232:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17232:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17232:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17232:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17232:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17240:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17240:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17300:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17300:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17258:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17258:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17258:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17258:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:17258:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:17232:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17232:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17232:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17232:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17232:mlx:core:mlx_fabric_open():172 libfabric:17232:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17232:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17232:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85942:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17258:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:17258:mlx:core:mlx_getinfo():301 Loaded MLX version 1.8.0 libfabric:17258:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:17258:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:17258:mlx:core:mlx_fabric_open():172 libfabric:17258:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:17258:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17258:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17267:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17267:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85865:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17274:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17274:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17261:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17261:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17313:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17313:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17259:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17259:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17307:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17307:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85924:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17242:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17242:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17252:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17252:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17221:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17221:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17268:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17268:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17271:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17271:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17317:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17317:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17291:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17314:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17314:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85945:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85945:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:17245:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17332:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17332:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85945:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:17236:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17236:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17225:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17225:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17279:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17279:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17234:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17234:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17260:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17260:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17277:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17277:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17315:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17315:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17328:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17328:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17238:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17238:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85916:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85945:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17270:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17270:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85923:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17341:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17341:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17249:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17249:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17338:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17338:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17231:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17231:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17232:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17232:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17235:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17235:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17239:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17239:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17258:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17258:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17276:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17276:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:17278:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:17278:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85945:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85941:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85992:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85992:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85945:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85992:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85945:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85945:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85945:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85992:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:17322:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85992:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85992:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85992:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85992:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85992:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17310:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17337:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17218:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17292:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17295:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85938:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85984:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85895:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17297:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17256:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17294:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17283:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85949:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85949:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85949:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85949:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85949:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:17274:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85949:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85944:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85949:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85949:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85949:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:17300:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17261:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17313:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17240:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85893:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17307:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17259:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17267:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17242:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85890:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17252:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17221:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17271:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85986:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17314:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17268:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85875:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17317:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85942:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:17270:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17332:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17238:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17260:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17236:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17234:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17225:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17277:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17328:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17341:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17279:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17232:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17239:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17231:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17235:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17249:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17258:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17276:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17278:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17315:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:17338:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [261]... libfabric:85865:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85865:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85865:core:mr:ofi_default_cache_size():78 default cache size=2103506784 libfabric:85880:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85880:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85880:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85956:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85977:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85880:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85880:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85880:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85880:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85880:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85880:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85924:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85940:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85866:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85952:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85952:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85952:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85952:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85967:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85965:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85965:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85965:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85965:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85889:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85925:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85920:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85923:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85950:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85950:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85950:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85950:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85936:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85957:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85957:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85957:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85957:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85982:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85916:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85916:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85916:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85916:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85907:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85916:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85975:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85946:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85916:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85916:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85916:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85916:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85935:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85930:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85963:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85888:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85941:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85941:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85941:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85941:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85941:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85941:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85941:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85941:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85941:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85938:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85928:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85984:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85895:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85881:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85991:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85908:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85902:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85961:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85873:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85903:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85948:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85944:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85913:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85886:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85912:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85897:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85890:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85909:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85909:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85909:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85909:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85893:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85986:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85945:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85992:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85992:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85945:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85910:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85911:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85911:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85911:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85911:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85875:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85904:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85870:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85945:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85992:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85901:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85945:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85945:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85992:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85992:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85945:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85992:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85949:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85949:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85942:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85942:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85942:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85942:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85942:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85942:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85942:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85942:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85942:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85956:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85878:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85878:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85878:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85878:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85945:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85949:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85992:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85949:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85949:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85949:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85977:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85905:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85905:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85905:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85905:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85962:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85962:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85962:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85962:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85960:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85960:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85960:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85960:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85983:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85983:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85983:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85983:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85973:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85973:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85973:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85973:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85877:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85943:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85943:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85943:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85943:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85945:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85945:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85945:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85945:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85992:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85992:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85992:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85992:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85866:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85949:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85967:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85940:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85924:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85924:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85924:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85924:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85924:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85925:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85924:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85924:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85924:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85924:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85889:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85992:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85992:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85992:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85992:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85992:mlx:core:mlx_fabric_open():172 libfabric:85992:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85992:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85992:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85945:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85945:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85945:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85945:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85945:mlx:core:mlx_fabric_open():172 libfabric:85945:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85945:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85945:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85949:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85949:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85949:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85949:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85920:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85919:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85919:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85919:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85919:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85947:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85947:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85947:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85947:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85980:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85980:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85980:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85980:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85966:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85936:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85990:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85975:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85923:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85923:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85923:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85923:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85880:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85880:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85923:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85923:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85923:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85923:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85923:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85982:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85949:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85949:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85949:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85949:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85949:mlx:core:mlx_fabric_open():172 libfabric:85949:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85949:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85949:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85892:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85892:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85892:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85892:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85884:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85884:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85884:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85884:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85978:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85978:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85978:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85978:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85907:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85946:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85935:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85930:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85880:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85888:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85880:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85880:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85880:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85963:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85883:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85883:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85883:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85883:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85891:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85891:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85891:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85891:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85921:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85921:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85921:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85921:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85959:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85959:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85959:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85959:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85974:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85974:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85974:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85974:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85979:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85979:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85979:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85979:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85903:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85928:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85991:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85908:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85881:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85873:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85902:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85880:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85961:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85913:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85938:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85938:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85948:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85897:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85938:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85938:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85886:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85868:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85868:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85868:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85868:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85938:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85912:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85984:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85984:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85964:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85964:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85964:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85964:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85984:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85984:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85922:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85922:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85922:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85922:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85938:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85876:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85876:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85876:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85876:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85938:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85938:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85938:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85984:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85898:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85898:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85898:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85898:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85895:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85895:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85887:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85887:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85887:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85887:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85895:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85984:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85895:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85984:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85984:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85984:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85895:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85880:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85880:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85880:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85880:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85895:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85895:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85895:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85895:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85916:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85916:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85944:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85944:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85944:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85944:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85944:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85890:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85890:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85944:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85944:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85944:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85944:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85890:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85893:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85893:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85890:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85893:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85890:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85893:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85904:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85893:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85870:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85910:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85890:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85893:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85890:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85890:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85890:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85893:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85893:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85893:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85986:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85986:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85986:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85986:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85867:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85867:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85867:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85867:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85916:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85916:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85916:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85916:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85986:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85941:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85941:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85874:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85874:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85874:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85874:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85915:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85915:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85915:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85915:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85969:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85969:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85969:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85969:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85986:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85986:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85986:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85986:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85872:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85872:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85872:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85872:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85900:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85900:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85900:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85900:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85875:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85875:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85901:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85875:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85918:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85918:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85918:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85918:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85875:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85879:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85879:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85879:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85879:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85875:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85914:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85914:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85914:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85914:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85875:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85992:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85992:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85875:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85875:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85875:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85917:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85917:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85917:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85917:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85945:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85945:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85871:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85871:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85871:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85871:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85951:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85951:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85951:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85951:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85970:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85970:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85970:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85970:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85916:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85880:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85880:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85880:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85880:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85880:mlx:core:mlx_fabric_open():172 libfabric:85880:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85880:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85880:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85941:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85941:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85941:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85941:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85933:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85933:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85933:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85933:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85976:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85976:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85976:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85976:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85934:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85934:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85934:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85934:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85953:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85953:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85953:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85953:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85885:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85885:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85885:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85885:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85916:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85916:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85916:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85916:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85981:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85981:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85981:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85981:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85906:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85906:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85906:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85906:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85894:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85894:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85894:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85894:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85968:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85968:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85968:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85968:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85958:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85958:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85958:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85958:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85956:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85956:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85926:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85926:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85926:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85926:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85869:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85869:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85869:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85869:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85927:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85927:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85927:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85927:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85956:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85939:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85939:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85939:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85939:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85896:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85896:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85896:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85896:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85949:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85949:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85955:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85955:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85955:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85955:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85956:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85956:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85971:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85971:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85971:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85971:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85941:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85985:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85985:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85985:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85985:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85956:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85956:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85956:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85956:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85987:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85987:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85987:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85987:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85877:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85937:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85937:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85937:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85937:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85899:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85899:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85899:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85899:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85882:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85882:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85882:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85882:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85916:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85916:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85916:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85916:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85916:mlx:core:mlx_fabric_open():172 libfabric:85916:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85916:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85916:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85988:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85988:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85988:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85988:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85989:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85989:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85989:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85989:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85929:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85929:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85929:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85929:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85954:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85954:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85954:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85954:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85972:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85972:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85972:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85972:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85931:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85931:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85931:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85931:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85932:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85932:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85932:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85932:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85977:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85977:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85977:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85977:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85977:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85977:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85977:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85977:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85977:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85866:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85866:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85967:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85967:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85866:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85967:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85941:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85941:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85941:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85941:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85866:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85940:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85940:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85967:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85940:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85866:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85967:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85940:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85942:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85942:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85940:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85866:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85966:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85967:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85866:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85866:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85866:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85967:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85967:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85967:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85889:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85889:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85925:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85925:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85940:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85889:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85940:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85940:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85940:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85925:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85889:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85990:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85925:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85889:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85925:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85889:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85889:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85889:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85889:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85925:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85920:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85920:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85925:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85925:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85925:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85920:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85920:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85941:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85941:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85941:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85920:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85941:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85941:mlx:core:mlx_fabric_open():172 libfabric:85941:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85941:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85941:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85975:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85975:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85920:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85975:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85920:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85920:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85920:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85936:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85936:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85975:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85936:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85975:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85936:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85936:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85975:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85982:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85982:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85975:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85975:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85975:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85936:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85942:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85982:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85936:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85936:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85936:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85942:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85942:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85942:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85982:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85982:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85907:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85907:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85907:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85982:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85946:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85946:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85907:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85982:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85982:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85982:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85946:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85907:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85946:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85930:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85930:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85935:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85935:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85946:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85930:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85935:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85930:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85935:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85888:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85888:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85930:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85935:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85963:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85963:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85942:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85888:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85963:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85907:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85888:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85963:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85907:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85907:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85907:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85903:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85903:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85963:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85888:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85991:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85991:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85881:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85881:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85908:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85908:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85873:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85873:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85903:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85946:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85913:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85913:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85991:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85902:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85902:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85908:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85930:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85886:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85886:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85948:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85948:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85961:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85961:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85881:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85897:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85897:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85912:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85912:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85946:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85946:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85946:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85873:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85903:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85930:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85930:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85930:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85928:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85928:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85908:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85924:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85924:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85991:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85886:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85902:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85948:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85961:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85873:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85913:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85881:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85897:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85912:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85928:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85935:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85903:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85935:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85935:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85935:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85902:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85908:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85991:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85886:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85961:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85873:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85881:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85912:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85948:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85897:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85904:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85904:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85963:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85913:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85928:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85870:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85870:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85886:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85902:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85961:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85910:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85910:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85948:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85963:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85963:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85963:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85904:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85912:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85897:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85928:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85888:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85870:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85913:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85888:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85888:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85888:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85904:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85910:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85870:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85904:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85910:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85870:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85910:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85942:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85942:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85942:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85901:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85901:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85942:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85901:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85877:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85877:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85901:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85881:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85877:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85901:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85881:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85881:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85881:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85877:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85877:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85903:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85903:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85903:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85903:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85873:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85908:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85873:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85873:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85873:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85908:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85908:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85908:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85961:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85961:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85961:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85961:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85991:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85942:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85942:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85942:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85942:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85942:mlx:core:mlx_fabric_open():172 libfabric:85942:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85942:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85942:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85991:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85991:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85991:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85966:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85966:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85990:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85990:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85897:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85966:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85897:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85897:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85897:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85990:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85886:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85886:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85886:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85886:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85966:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85948:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85910:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85948:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85948:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85948:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85990:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85902:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85902:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85902:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85902:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85910:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85910:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85910:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85966:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85990:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85912:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85928:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85904:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85912:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85912:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85912:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85928:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85928:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85928:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85904:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85904:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85904:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85901:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85924:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85901:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85901:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85901:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85923:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85923:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85870:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85913:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85865:verbs:fabric:verbs_devs_print():883 list of verbs devices found for FI_EP_MSG: libfabric:85865:verbs:fabric:verbs_devs_print():887 #1 mlx5_0 - IPoIB addresses: libfabric:85865:verbs:fabric:verbs_devs_print():897 10.205.73.135 libfabric:85865:verbs:fabric:verbs_devs_print():897 fe80::ba59:9f03:d2:2 libfabric:85924:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85924:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85870:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85870:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85870:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85913:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85913:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85913:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85877:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85877:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85877:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85877:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85924:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85966:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85990:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85966:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85966:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85966:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85990:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85990:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85990:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85924:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85924:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85924:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85924:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85945:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85923:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85923:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85923:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85924:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85923:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85924:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85924:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85924:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85924:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85924:mlx:core:mlx_fabric_open():172 libfabric:85924:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85924:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85924:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85923:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85952:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85992:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85923:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85923:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85923:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85949:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85923:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85952:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85923:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85923:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85923:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85923:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85923:mlx:core:mlx_fabric_open():172 libfabric:85923:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85923:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85923:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85941:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85941:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85938:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85938:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85916:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85916:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85984:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85984:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85880:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85880:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85895:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85895:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85965:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85950:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85944:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85944:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85952:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85952:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85957:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85952:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85952:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85952:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85893:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85893:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85890:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85890:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85938:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85938:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85938:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85984:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85984:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85984:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85952:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85938:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85965:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85950:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85952:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85952:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85952:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85984:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85938:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85986:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85986:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85984:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85942:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85942:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85957:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85895:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85895:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85895:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85938:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85938:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85938:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85944:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85893:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85984:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85984:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85984:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85944:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85944:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85890:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85875:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85875:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85895:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85893:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85893:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85890:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85890:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85938:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85984:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85965:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85965:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85944:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85950:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85950:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85895:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85965:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85893:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85890:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85950:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85965:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85938:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85938:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85938:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85938:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85938:mlx:core:mlx_fabric_open():172 libfabric:85938:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85938:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85938:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85950:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85965:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85950:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85984:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85984:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85984:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85984:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85984:mlx:core:mlx_fabric_open():172 libfabric:85984:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85984:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85984:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85944:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85893:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85890:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85965:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85950:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85965:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85965:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85965:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85950:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85950:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85950:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85957:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85957:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85957:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85957:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85957:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85895:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85895:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85895:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85986:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85986:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85986:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85944:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85944:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85944:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85957:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85895:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85957:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85957:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85957:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85924:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85924:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85893:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85893:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85893:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85890:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85890:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85890:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85986:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85944:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85875:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85895:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85895:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85895:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85895:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85895:mlx:core:mlx_fabric_open():172 libfabric:85895:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85895:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85895:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85875:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85875:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85890:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85986:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85893:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85944:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85944:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85944:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85944:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85944:mlx:core:mlx_fabric_open():172 libfabric:85944:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85944:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85944:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85875:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85890:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85890:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85890:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85890:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85890:mlx:core:mlx_fabric_open():172 libfabric:85890:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85890:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85890:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85893:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85893:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85893:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85893:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85893:mlx:core:mlx_fabric_open():172 libfabric:85893:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85893:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85893:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85875:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85986:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85986:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85986:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85986:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85875:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85875:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85875:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85986:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85986:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85986:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85986:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85986:mlx:core:mlx_fabric_open():172 libfabric:85986:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85986:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85986:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85923:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85923:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85956:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85875:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85956:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85909:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85875:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85875:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85875:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85875:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85875:mlx:core:mlx_fabric_open():172 libfabric:85875:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85875:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85875:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85977:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85977:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85941:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85909:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85956:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85956:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85956:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85956:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85866:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85866:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85967:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85967:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85940:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85940:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85916:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85889:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85889:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85984:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85984:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85938:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85938:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85880:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85911:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85956:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85925:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85925:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85977:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85977:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85977:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85920:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85920:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85977:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85942:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85866:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85866:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85866:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85967:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85940:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85944:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85944:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85975:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85975:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85967:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85967:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85940:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85940:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85866:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85936:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85940:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85967:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85895:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85895:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85936:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85890:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85890:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85977:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85956:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85956:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85956:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85967:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85940:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85866:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85909:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85909:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85893:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85893:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85909:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85956:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85909:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85982:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85982:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85909:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85956:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85956:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85956:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85956:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85956:mlx:core:mlx_fabric_open():172 libfabric:85956:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85956:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85956:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85909:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85909:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85909:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85909:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85977:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85977:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85977:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85889:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85889:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85889:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85967:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85967:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85967:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85866:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85866:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85866:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85940:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85940:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85940:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85925:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85925:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85925:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85977:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85940:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85967:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85866:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85889:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85925:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85986:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85986:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85920:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85920:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85920:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85962:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85975:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85920:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85875:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85875:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85936:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85975:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85975:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85936:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85936:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85977:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85977:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85977:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85977:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85977:mlx:core:mlx_fabric_open():172 libfabric:85977:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85977:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85977:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85866:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85866:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85866:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85866:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85940:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85940:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85940:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85940:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85940:mlx:core:mlx_fabric_open():172 libfabric:85940:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85940:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85940:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85967:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85967:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85967:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85967:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85967:mlx:core:mlx_fabric_open():172 libfabric:85967:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85967:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85967:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85866:mlx:core:mlx_fabric_open():172 libfabric:85866:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85866:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85866:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85925:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85924:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85975:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85936:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85889:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85920:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85963:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85935:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85935:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85963:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85907:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85907:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85930:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85946:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85946:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85930:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85936:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85888:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85888:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85975:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85925:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85925:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85925:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85889:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85889:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85889:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85881:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85925:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85881:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85982:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85982:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85982:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85960:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85889:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85982:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85920:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85920:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85920:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85911:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85973:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85925:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85925:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85925:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85925:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85925:mlx:core:mlx_fabric_open():172 libfabric:85925:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85925:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85925:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85936:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85936:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85936:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85975:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85975:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85975:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85920:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85889:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85889:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85889:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85889:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85889:mlx:core:mlx_fabric_open():172 libfabric:85889:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85889:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85889:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85982:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85943:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85963:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85935:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85963:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85963:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85935:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85935:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85936:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85963:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85975:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85935:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85923:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85982:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85982:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85982:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85936:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85936:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85936:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85975:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85975:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85975:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85920:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85920:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85920:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85920:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85920:mlx:core:mlx_fabric_open():172 libfabric:85920:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85920:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85920:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85936:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85936:mlx:core:mlx_fabric_open():172 libfabric:85936:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85936:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85936:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85975:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85975:mlx:core:mlx_fabric_open():172 libfabric:85975:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85975:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85975:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85935:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85963:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85904:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85908:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85908:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85961:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85961:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85982:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85991:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85991:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85870:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85870:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85873:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85873:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85901:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85901:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85902:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85904:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85886:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85886:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85897:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85897:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85902:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85910:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85913:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85966:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85966:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85990:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85877:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85910:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85912:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85912:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85948:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85948:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85990:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85877:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85903:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85903:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85913:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85928:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85983:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85928:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85946:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85881:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85888:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85907:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85930:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85946:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85946:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85881:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85881:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85888:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85888:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85930:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85930:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85907:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85907:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85888:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85930:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85946:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85881:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85982:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85982:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85982:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85982:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85982:mlx:core:mlx_fabric_open():172 libfabric:85982:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85982:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85982:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85907:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85935:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85935:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85935:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85963:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85963:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85963:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85930:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85946:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85888:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85881:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85935:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85963:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85907:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85962:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85905:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85956:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85956:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85908:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85928:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85908:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85908:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85928:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85928:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85901:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85866:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85866:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85886:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85902:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85904:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85947:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85903:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85967:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85967:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85897:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85961:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85870:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85873:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85963:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85963:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85963:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85963:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85963:mlx:core:mlx_fabric_open():172 libfabric:85963:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85963:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85963:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85901:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85901:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85948:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85886:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85886:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85902:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85902:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85904:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85904:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85991:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85897:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85897:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85903:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85903:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85910:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85912:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85961:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85961:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85870:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85870:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85873:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85873:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85877:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85908:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85935:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85935:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85935:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85935:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85935:mlx:core:mlx_fabric_open():172 libfabric:85935:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85935:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85935:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85948:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85948:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85966:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85990:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85991:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85991:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85940:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85940:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85928:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85910:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85910:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85912:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85912:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85990:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85990:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85877:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85877:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85913:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85966:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85966:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85913:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85913:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85901:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85948:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85930:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85930:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85930:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85946:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85946:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85946:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85873:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85902:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85888:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85888:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85888:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85881:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85881:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85881:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85961:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85897:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85991:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85904:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85907:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85907:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85907:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85903:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85870:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85984:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85886:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85966:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85990:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85938:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85928:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85912:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85877:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85908:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85930:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85910:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85946:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85881:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85888:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85907:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85913:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85944:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85895:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85977:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85977:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85901:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85948:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85928:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85928:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85928:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85928:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85873:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85908:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85908:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85908:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85961:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85908:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85903:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85902:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85991:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85897:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85966:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85870:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85904:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85886:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85990:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85910:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85912:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85877:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85881:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85881:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85881:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85881:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85881:mlx:core:mlx_fabric_open():172 libfabric:85881:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85881:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85881:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85930:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85930:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85930:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85930:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85930:mlx:core:mlx_fabric_open():172 libfabric:85930:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85930:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85930:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85946:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85946:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85946:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85946:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85946:mlx:core:mlx_fabric_open():172 libfabric:85946:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85946:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85946:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85888:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85888:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85888:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85888:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85888:mlx:core:mlx_fabric_open():172 libfabric:85888:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85888:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85888:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85907:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85907:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85907:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85907:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85907:mlx:core:mlx_fabric_open():172 libfabric:85907:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85907:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85907:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85913:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85959:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85960:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85928:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85928:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85928:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85928:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85928:mlx:core:mlx_fabric_open():172 libfabric:85928:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85928:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85928:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85908:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85908:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85908:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85908:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85908:mlx:core:mlx_fabric_open():172 libfabric:85908:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85908:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85908:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85973:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85948:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85948:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85948:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85901:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85901:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85901:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85980:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85990:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85990:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85990:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85991:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85991:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85991:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85966:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85966:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85966:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85948:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85911:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85911:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85901:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85911:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85990:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85873:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85873:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85873:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85952:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85991:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85952:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85943:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85961:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85961:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85961:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85966:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85911:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85873:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85911:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85961:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85978:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85902:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85902:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85902:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85903:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85903:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85903:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85904:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85904:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85904:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85870:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85870:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85870:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85897:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85897:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85897:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85877:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85877:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85877:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85910:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85910:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85910:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85912:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85912:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85912:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85886:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85886:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85886:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85911:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85911:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85911:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85911:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85986:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85897:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85913:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85913:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85913:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85886:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85870:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85903:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85877:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85893:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85902:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85904:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85910:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85912:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85875:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85890:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85913:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85889:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85889:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85925:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85925:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85948:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85948:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85948:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85948:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85948:mlx:core:mlx_fabric_open():172 libfabric:85948:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85948:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85948:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85901:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85901:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85901:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85901:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85901:mlx:core:mlx_fabric_open():172 libfabric:85901:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85901:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85901:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85991:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85991:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85991:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85991:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85991:mlx:core:mlx_fabric_open():172 libfabric:85991:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85991:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85991:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85990:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85990:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85990:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85990:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85990:mlx:core:mlx_fabric_open():172 libfabric:85990:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85990:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85990:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85961:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85961:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85961:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85961:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85961:mlx:core:mlx_fabric_open():172 libfabric:85961:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85961:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85961:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85936:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85936:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85975:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85975:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85873:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85920:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85920:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85966:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85873:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85873:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85873:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85873:mlx:core:mlx_fabric_open():172 libfabric:85873:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85873:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85873:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85966:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85966:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85966:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85966:mlx:core:mlx_fabric_open():172 libfabric:85966:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85966:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85966:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85982:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85982:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85897:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85897:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85897:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85897:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85897:mlx:core:mlx_fabric_open():172 libfabric:85897:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85897:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85897:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85886:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85886:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85886:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85886:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85886:mlx:core:mlx_fabric_open():172 libfabric:85886:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85886:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85886:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85870:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85870:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85870:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85870:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85870:mlx:core:mlx_fabric_open():172 libfabric:85870:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85870:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85870:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85983:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85877:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85903:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85903:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85903:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85877:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85877:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85877:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85877:mlx:core:mlx_fabric_open():172 libfabric:85877:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85877:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85877:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85903:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85903:mlx:core:mlx_fabric_open():172 libfabric:85903:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85903:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85903:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85902:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85904:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85904:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85904:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85904:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85912:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85912:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85912:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85912:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85912:mlx:core:mlx_fabric_open():172 libfabric:85965:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85902:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85902:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85902:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85902:mlx:core:mlx_fabric_open():172 libfabric:85902:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85902:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85902:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85904:mlx:core:mlx_fabric_open():172 libfabric:85904:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85904:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85904:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85912:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85912:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85912:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85965:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85910:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85910:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85910:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85910:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85910:mlx:core:mlx_fabric_open():172 libfabric:85910:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85910:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85910:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85950:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85913:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85913:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85913:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85950:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85913:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85913:mlx:core:mlx_fabric_open():172 libfabric:85913:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85913:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85913:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85919:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85935:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85935:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85963:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85963:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85952:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85952:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85952:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85952:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85957:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85957:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85974:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85892:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85950:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85950:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85950:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85950:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85965:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85965:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85965:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85965:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85957:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85884:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85957:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85957:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85957:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85952:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85891:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85952:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85952:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85952:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85952:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85979:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85964:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85950:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85965:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85905:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85957:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85956:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85883:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85947:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85967:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85940:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85930:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85930:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85866:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85946:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85946:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85888:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85888:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85907:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85907:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85881:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85881:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85962:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85962:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85962:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85965:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85965:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85965:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85965:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85962:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85957:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85957:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85957:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85962:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85957:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85950:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85950:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85950:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85950:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85952:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85952:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85952:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85952:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85952:mlx:core:mlx_fabric_open():172 libfabric:85952:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85952:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85952:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85962:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85962:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85962:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85962:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85878:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85977:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85928:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85928:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85909:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85909:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85908:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85908:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85909:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85909:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85909:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85909:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85965:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85965:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85965:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85965:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85965:mlx:core:mlx_fabric_open():172 libfabric:85965:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85965:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85965:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85950:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85950:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85950:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85950:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85950:mlx:core:mlx_fabric_open():172 libfabric:85950:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85950:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85950:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85957:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85957:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85957:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85957:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85957:mlx:core:mlx_fabric_open():172 libfabric:85957:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85957:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85957:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85921:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85948:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85948:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85909:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85961:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85961:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85975:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85889:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85925:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85991:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85991:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85990:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85990:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85959:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85901:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85901:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85936:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85966:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85966:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85920:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85897:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85897:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85982:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85873:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85873:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85898:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85909:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85909:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85909:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85909:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85886:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85886:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85903:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85903:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85877:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85877:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85902:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85902:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85912:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85912:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85870:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85870:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85910:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85910:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85904:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85904:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85913:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85913:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85980:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85969:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85935:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85922:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85963:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85960:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85960:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85960:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85960:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85960:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85887:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85960:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85960:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85960:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85960:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85911:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85911:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85978:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85973:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85973:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85973:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85973:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85909:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85909:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85909:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85909:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85909:mlx:core:mlx_fabric_open():172 libfabric:85909:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85909:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85909:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85973:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85973:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85973:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85973:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85973:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85951:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85911:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85911:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85911:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85911:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85876:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85943:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85943:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85943:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85943:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85943:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85943:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85943:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85943:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85943:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85970:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85976:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85911:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85933:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85919:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85872:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85930:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85946:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85981:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85871:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85934:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85953:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85955:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85867:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85915:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85881:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85888:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85907:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85900:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85874:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85914:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85931:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85974:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85958:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85879:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85968:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85952:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85952:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85985:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85983:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85983:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85929:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85983:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85983:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85928:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85932:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85954:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85983:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85892:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85937:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85971:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85983:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85983:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85983:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85983:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85908:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85911:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85911:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85911:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85884:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85911:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85917:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85868:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85918:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85885:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85972:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85950:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85950:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85939:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85965:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85965:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85988:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85957:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85957:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85894:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85891:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85906:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85989:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85869:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85979:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85896:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85987:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85948:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85964:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85882:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85911:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85911:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85911:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85911:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85911:mlx:core:mlx_fabric_open():172 libfabric:85911:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85911:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85911:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85899:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85962:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85962:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85927:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85883:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85961:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85926:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85990:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85991:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85901:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85873:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85897:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85903:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85877:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85878:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85966:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85962:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85962:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85962:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85870:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85886:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85902:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85962:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85910:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85912:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85905:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85905:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85905:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85904:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85913:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85905:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85905:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85947:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85947:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85947:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85905:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85947:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85905:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85905:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85905:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85947:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85947:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85947:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85947:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85947:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85909:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85909:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85962:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85921:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85962:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85962:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85962:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85962:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85898:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85962:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85962:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85962:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85962:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85962:mlx:core:mlx_fabric_open():172 libfabric:85962:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85962:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85962:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85969:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85960:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85960:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85922:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85959:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85959:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85959:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85959:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85959:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85887:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85959:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85959:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85959:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85959:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85952:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85973:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85973:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85960:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85980:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85980:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85960:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85960:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85960:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85980:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85980:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85980:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85951:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85950:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85980:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85965:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85980:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85980:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85980:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85876:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85978:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85978:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85957:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85978:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85978:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85978:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85978:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85978:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85978:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85978:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85943:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85943:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85970:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85911:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85911:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85973:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85973:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85973:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85973:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85960:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85933:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85976:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85871:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85953:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85981:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85955:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85867:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85934:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85915:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85900:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85973:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85872:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85914:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85931:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85958:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85879:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85960:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85960:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85960:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85960:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85985:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85874:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85968:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85929:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85943:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85943:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85943:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85943:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85932:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85954:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85937:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85971:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85919:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85919:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85868:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85974:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85974:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85919:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85909:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85974:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85988:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85972:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85973:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85973:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85973:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85973:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85919:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85974:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85919:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85917:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85974:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85918:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85939:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85960:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85960:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85960:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85960:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85960:mlx:core:mlx_fabric_open():172 libfabric:85960:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85960:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85960:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85983:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85983:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85885:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85919:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85974:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85919:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85919:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85919:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85974:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85974:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85974:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85906:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85943:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85892:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85892:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85894:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85896:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85892:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85884:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85884:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85987:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85892:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85869:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85884:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85892:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85884:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85884:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85989:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85892:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85884:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85892:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85892:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85892:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85973:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85973:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85973:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85973:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85973:mlx:core:mlx_fabric_open():172 libfabric:85973:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85973:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85973:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85884:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85884:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85884:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85891:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85891:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85899:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85882:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85891:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85943:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85943:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85943:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85943:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85891:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85891:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85962:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85962:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85927:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85891:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85926:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85891:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85891:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85891:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85983:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85983:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85983:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85983:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85979:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85979:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85979:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85979:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85979:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85964:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85964:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85964:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85979:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85964:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85979:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85979:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85979:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85964:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85964:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85964:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85964:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85964:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85943:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85943:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85943:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85943:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85943:mlx:core:mlx_fabric_open():172 libfabric:85943:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85943:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85943:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85883:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85883:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85883:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85883:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85883:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85983:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85883:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85883:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85883:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85883:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85878:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85878:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85878:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85878:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85878:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85905:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85905:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85878:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85878:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85878:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85878:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85983:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85983:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85983:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85983:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85947:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85947:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85911:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85905:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85905:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85905:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85905:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85983:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85983:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85983:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85983:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85983:mlx:core:mlx_fabric_open():172 libfabric:85983:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85983:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85983:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85947:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85947:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85947:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85947:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85921:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85921:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85921:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85921:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85921:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85921:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85921:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85921:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85921:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85905:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85947:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85960:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85960:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85898:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85898:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85898:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85969:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85969:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85898:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85969:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85898:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85969:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85969:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85922:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85922:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85922:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85922:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85898:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85969:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85922:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85887:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85887:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85898:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85898:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85898:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85969:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85969:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85969:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85947:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85947:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85947:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85887:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85947:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85922:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85922:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85922:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85922:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85887:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85905:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85905:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85905:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85887:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85905:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85973:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85973:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85887:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85887:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85887:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85887:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85962:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85943:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85943:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85947:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85947:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85947:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85947:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85947:mlx:core:mlx_fabric_open():172 libfabric:85947:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85947:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85947:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85951:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85951:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85876:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85876:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85951:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85876:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85905:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85905:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85905:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85905:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85951:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85905:mlx:core:mlx_fabric_open():172 libfabric:85905:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85905:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85905:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85876:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85951:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85876:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85959:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85959:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85865:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85970:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85970:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85970:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85951:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85876:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85970:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85976:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85976:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85951:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85951:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85951:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85876:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85876:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85876:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85970:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85976:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85958:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85958:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85976:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85933:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85933:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85934:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85934:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85970:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85958:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85976:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85970:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85970:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85970:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85933:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85934:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85958:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85871:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85871:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85955:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85955:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85914:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85914:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85915:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85915:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85981:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85981:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85872:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85872:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85933:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85934:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85953:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85953:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85958:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85871:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85900:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85900:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85914:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85915:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85953:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85955:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85985:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85985:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85867:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85867:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85933:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85934:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85976:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85981:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85872:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85879:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85879:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85871:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85900:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85976:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85976:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85976:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85914:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85915:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85932:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85932:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85937:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85937:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85953:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85955:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85981:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85931:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85931:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85985:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85867:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85872:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85879:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85900:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85954:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85954:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85968:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85968:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85874:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85874:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85871:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85958:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85929:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85929:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85914:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85915:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85932:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85937:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85953:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85955:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85958:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85958:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85958:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85981:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85931:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85985:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85867:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85879:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85954:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85872:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85900:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85968:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85971:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85971:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85874:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85929:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85934:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85937:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85933:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85931:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85932:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85933:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85933:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85933:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85934:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85934:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85934:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85954:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85968:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85879:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85939:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85939:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85985:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85867:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85874:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85929:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85988:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85988:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85917:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85917:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85937:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85971:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85972:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85972:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85931:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85932:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85954:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85868:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85868:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85939:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85918:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85918:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85968:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85980:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85980:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85874:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85885:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85885:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85906:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85906:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85929:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85971:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85988:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85917:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85972:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85983:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85983:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85869:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85869:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85939:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85868:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85906:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85918:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85987:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85987:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85988:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85989:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85989:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85885:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85894:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85894:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85896:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85896:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85971:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85917:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85972:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85869:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85899:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85899:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85906:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85939:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85868:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85918:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85988:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85885:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85987:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85882:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85882:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85869:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85894:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85896:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85906:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85917:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85972:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85989:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85899:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85926:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85926:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85927:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85927:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85918:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85871:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85987:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85868:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85885:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85882:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85981:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85869:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85894:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85896:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85871:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85871:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85871:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85899:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85981:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85981:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85981:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85989:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85926:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85927:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85987:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85882:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85894:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85896:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85899:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85926:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85927:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85882:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85989:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85947:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85947:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85926:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85927:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85959:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85960:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85905:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85905:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85959:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85959:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85959:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85959:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85978:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85978:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85955:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85953:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85955:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85955:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85955:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85953:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85953:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85953:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85943:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85865:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85973:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85914:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85915:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85914:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85914:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85914:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85915:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85915:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85915:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85872:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85900:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85872:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85872:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85872:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85900:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85900:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85900:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85985:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85985:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85985:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85985:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85879:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85879:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85879:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85879:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85959:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85959:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85959:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85959:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85959:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85959:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85959:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85959:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85959:mlx:core:mlx_fabric_open():172 libfabric:85959:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85959:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85959:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85867:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85867:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85867:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85867:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85937:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85932:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85931:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85937:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85937:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85937:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85932:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85932:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85932:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85954:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85931:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85931:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85931:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85954:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85954:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85954:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85968:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85980:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85874:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85968:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85968:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85968:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85929:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85874:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85874:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85874:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85971:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85929:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85929:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85929:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85980:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85980:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85980:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85939:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85971:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85971:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85971:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85988:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85939:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85939:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85939:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85917:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85988:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85988:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85988:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85917:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85917:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85917:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85972:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85972:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85972:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85972:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85906:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85918:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85906:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85906:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85906:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85868:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85918:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85918:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85918:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85885:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85865:verbs:fabric:vrb_get_device_attrs():618 device mlx5_0: first found active port is 1 libfabric:85865:core:core:ofi_register_provider():474 registering provider: verbs (113.20) libfabric:85868:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85868:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85868:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85869:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85987:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85885:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85885:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85885:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85978:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85980:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85869:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85869:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85869:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85894:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85987:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85987:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85987:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85978:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85978:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85978:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85865:core:core:ofi_register_provider():474 registering provider: tcp (113.20) libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85896:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85894:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85894:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85894:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85899:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85896:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85896:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85896:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85882:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85899:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85899:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85899:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85865:core:core:ofi_register_provider():474 registering provider: sockets (113.20) libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85989:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85882:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85882:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85882:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85978:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85926:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85927:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85989:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85989:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85989:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85926:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85926:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85926:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85927:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85927:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85927:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ZE not supported libfabric:85865:core:core:ofi_register_provider():474 registering provider: shm (113.20) libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_CUDA not supported libfabric:85865:core:core:ofi_hmem_init():209 Hmem iface FI_HMEM_ROCR not supported libfabric:85865:core:core:ze_hmem_dl_init():422 Failed to dlopen libze_loader.so libfabric:85865:core:core:ofi_hmem_init():214 Failed to initialize hmem iface FI_HMEM_ZE: No data available libfabric:85865:core:core:ofi_register_provider():474 registering provider: ofi_rxm (113.20) libfabric:85983:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85959:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85959:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85980:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85980:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85980:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85980:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85980:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85980:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85980:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85980:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85980:mlx:core:mlx_fabric_open():172 libfabric:85980:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85980:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85980:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85978:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85978:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85978:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85978:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85947:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85978:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85978:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85978:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85978:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85978:mlx:core:mlx_fabric_open():172 libfabric:85978:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85978:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85978:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85919:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85919:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85974:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85974:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85905:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85978:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85978:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85980:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85980:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85959:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85892:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85892:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85884:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85884:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85919:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85919:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85919:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85919:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85974:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85919:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85974:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85974:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85974:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85891:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85891:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85974:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85978:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85980:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85979:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85979:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85919:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85919:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85919:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85919:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85892:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85919:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85919:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85919:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85919:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85919:mlx:core:mlx_fabric_open():172 libfabric:85919:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85919:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85919:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85892:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85892:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85974:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85974:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85974:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85974:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85892:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85964:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85964:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85884:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85974:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85974:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85974:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85974:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85974:mlx:core:mlx_fabric_open():172 libfabric:85974:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85974:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85974:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85892:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85884:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85884:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85884:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85884:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85891:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85891:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85891:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85891:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85891:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85892:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85892:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85892:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85892:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85884:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85884:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85884:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85884:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85892:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85892:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85892:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85892:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85892:mlx:core:mlx_fabric_open():172 libfabric:85892:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85892:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85892:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85884:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85884:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85884:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85884:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85884:mlx:core:mlx_fabric_open():172 libfabric:85884:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85884:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85884:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85919:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85919:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85974:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85974:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85979:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85979:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85979:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85979:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85979:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85883:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85883:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85964:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85964:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85964:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85964:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85891:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85891:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85891:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85891:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85964:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85891:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85891:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85891:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85891:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85891:mlx:core:mlx_fabric_open():172 libfabric:85891:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85891:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85891:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85979:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85979:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85979:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85979:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85979:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85979:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85979:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85979:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85979:mlx:core:mlx_fabric_open():172 libfabric:85979:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85979:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85979:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85892:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85892:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85884:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85884:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85878:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85878:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85883:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85964:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85964:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85964:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85964:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85883:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85883:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85883:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85964:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85964:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85964:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85964:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85964:mlx:core:mlx_fabric_open():172 libfabric:85964:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85964:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85964:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85883:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85891:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85891:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85974:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85919:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85979:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85979:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85883:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85883:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85883:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85883:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85883:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85883:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85883:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85883:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85883:mlx:core:mlx_fabric_open():172 libfabric:85883:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85883:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85883:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85964:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85964:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85878:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85878:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85878:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85878:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85878:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85884:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85892:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85891:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85883:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85883:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85878:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85878:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85878:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85878:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85878:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85878:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85878:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85878:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85878:mlx:core:mlx_fabric_open():172 libfabric:85878:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85878:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85878:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85979:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85921:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85921:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85964:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85878:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85878:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85883:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85921:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85921:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85921:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85921:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85898:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85898:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85921:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85969:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85969:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85922:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85922:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85878:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85887:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85887:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85921:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85921:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85921:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85921:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85921:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85921:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85921:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85921:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85921:mlx:core:mlx_fabric_open():172 libfabric:85921:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85921:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85921:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85898:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85922:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85898:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85898:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85898:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85922:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85922:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85922:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85969:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85898:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85969:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85969:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85969:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85921:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85921:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85922:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85969:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85887:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85887:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85887:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85887:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85951:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85951:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85887:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85876:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85876:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85970:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85970:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85898:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85898:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85898:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85898:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85922:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85922:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85922:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85922:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85976:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85976:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85969:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85969:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85969:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85969:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85898:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85898:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85898:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85898:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85898:mlx:core:mlx_fabric_open():172 libfabric:85898:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85898:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85898:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85922:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85922:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85922:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85922:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85922:mlx:core:mlx_fabric_open():172 libfabric:85922:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85922:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85922:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85969:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85969:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85969:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85969:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85969:mlx:core:mlx_fabric_open():172 libfabric:85969:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85969:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85969:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85958:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85958:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85887:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85887:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85887:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85887:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85933:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85933:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85887:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85887:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85887:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85887:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85887:mlx:core:mlx_fabric_open():172 libfabric:85887:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85887:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85887:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85934:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85934:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85871:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85871:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85981:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85981:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85921:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85876:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85876:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85876:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85876:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85951:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85876:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85951:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85951:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85951:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85951:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85922:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85922:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85898:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85898:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85969:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85969:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85953:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85953:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85955:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85955:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85976:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85976:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85976:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85976:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85887:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85887:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85915:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85915:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85914:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85976:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85914:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85872:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85872:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85970:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85900:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85900:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85970:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85970:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85970:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85970:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85879:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85985:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85879:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85985:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85876:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85876:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85876:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85876:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85876:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85876:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85876:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85876:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85876:mlx:core:mlx_fabric_open():172 libfabric:85876:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85876:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85876:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85951:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85951:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85951:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85951:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85951:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85951:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85951:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85951:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85951:mlx:core:mlx_fabric_open():172 libfabric:85951:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85951:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85951:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85867:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85867:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85976:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85976:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85976:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85976:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85933:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85934:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85958:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85976:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85976:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85976:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85976:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85976:mlx:core:mlx_fabric_open():172 libfabric:85976:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85976:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85976:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85933:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85933:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85933:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85934:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85934:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85934:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85958:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85958:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85958:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85937:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85937:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85871:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85932:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85932:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85871:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85871:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85871:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85933:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85934:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85954:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85954:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85958:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85931:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85931:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85968:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85971:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85929:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85929:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85968:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85971:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85871:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85981:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85939:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85874:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85939:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85970:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85970:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85970:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85988:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85988:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85874:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85917:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85970:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85917:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85981:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85981:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85981:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85972:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85972:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85872:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85885:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85885:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85900:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85906:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85906:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85914:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85915:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85918:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85918:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85953:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85955:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85970:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85970:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85970:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85970:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85970:mlx:core:mlx_fabric_open():172 libfabric:85970:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85970:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85970:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85981:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85868:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85868:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85869:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85869:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85872:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85872:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85872:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85953:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85953:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85955:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85955:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85900:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85900:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85953:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85955:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85914:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85914:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85914:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85915:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85915:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85915:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85872:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85900:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85987:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85987:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85882:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85882:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85894:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85894:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85955:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85899:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85953:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85899:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85900:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85914:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85915:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85926:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85926:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85989:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85989:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85896:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85896:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85927:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85927:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85933:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85933:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85933:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85934:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85934:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85934:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85958:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85958:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85958:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85933:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85934:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85958:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85871:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85871:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85871:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85871:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85876:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85876:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85933:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85933:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85933:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85933:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85933:mlx:core:mlx_fabric_open():172 libfabric:85933:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85933:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85933:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85934:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85934:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85934:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85934:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85934:mlx:core:mlx_fabric_open():172 libfabric:85934:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85934:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85934:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85951:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85951:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85958:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85958:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85958:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85958:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85958:mlx:core:mlx_fabric_open():172 libfabric:85958:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85958:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85958:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85871:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85871:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85871:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85871:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85871:mlx:core:mlx_fabric_open():172 libfabric:85871:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85871:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85871:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85898:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85969:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85976:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85976:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85922:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85887:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85981:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85981:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85981:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85981:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85872:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85872:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85872:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85872:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85981:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85981:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85981:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85981:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85981:mlx:core:mlx_fabric_open():172 libfabric:85981:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85981:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85981:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85872:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85872:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85872:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85872:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85872:mlx:core:mlx_fabric_open():172 libfabric:85872:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85872:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85872:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85953:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85953:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85953:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85955:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85955:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85955:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85955:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85953:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85900:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85900:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85900:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85914:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85914:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85914:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85915:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85915:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85915:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85900:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85914:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85915:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85879:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85970:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85970:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85985:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85879:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85879:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85879:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85955:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85955:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85955:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85955:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85955:mlx:core:mlx_fabric_open():172 libfabric:85955:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85955:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85955:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85953:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85953:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85953:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85953:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85953:mlx:core:mlx_fabric_open():172 libfabric:85953:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85953:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85953:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85985:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85985:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85985:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85867:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85900:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85900:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85900:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85900:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85900:mlx:core:mlx_fabric_open():172 libfabric:85900:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85900:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85900:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85914:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85914:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85914:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85914:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85914:mlx:core:mlx_fabric_open():172 libfabric:85914:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85914:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85914:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85915:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85915:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85915:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85915:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85915:mlx:core:mlx_fabric_open():172 libfabric:85915:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85915:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85915:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85865:psm3:core:fi_prov_ini():680 build options: VERSION=1101.0=11.1.0.0, HAVE_PSM3_SRC=1, PSM3_CUDA=0 libfabric:85865:core:core:ofi_register_provider():474 registering provider: psm3 (1101.0) libfabric:85879:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85867:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85867:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85867:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85985:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85867:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85937:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85932:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85954:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85931:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85874:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85937:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85937:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85937:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85954:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85954:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85954:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85929:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85932:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85932:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85932:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85971:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85917:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85931:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85931:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85931:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85939:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85874:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85874:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85918:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85988:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85874:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85929:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85929:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85929:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85868:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85869:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85968:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85971:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85971:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85971:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85917:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85917:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85917:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85939:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85939:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85939:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85926:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85988:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85988:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85869:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85869:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85906:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85918:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85918:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85918:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85988:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85868:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85868:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85868:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85968:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85968:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85968:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85869:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85885:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85899:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85926:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85926:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85926:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85894:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85906:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85906:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85906:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85972:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85987:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85896:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85882:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85885:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85885:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85885:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85899:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85899:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85899:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85937:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85894:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85894:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85894:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85987:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85987:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85987:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85896:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85896:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85896:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85954:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85932:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85972:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85972:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85972:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85882:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85882:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85882:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85931:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85989:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85989:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85989:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85989:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85874:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85929:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85917:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85971:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85939:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85879:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85879:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85879:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85879:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85918:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85868:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85968:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85926:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85906:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85985:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85985:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85985:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85985:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85869:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85988:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85885:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85899:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85927:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85972:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85987:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85896:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85882:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85894:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85867:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85867:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85867:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85867:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85927:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85927:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85927:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85989:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85879:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85879:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85879:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85879:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85879:mlx:core:mlx_fabric_open():172 libfabric:85879:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85879:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85879:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85985:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85985:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85985:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85985:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85985:mlx:core:mlx_fabric_open():172 libfabric:85985:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85985:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85985:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85867:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85867:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85867:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85867:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85867:mlx:core:mlx_fabric_open():172 libfabric:85867:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85867:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85867:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85927:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85937:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85937:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85937:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85937:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85932:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85932:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85932:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85954:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85954:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85954:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85932:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85954:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85931:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85931:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85931:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85931:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85958:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85958:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85933:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85933:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85934:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85934:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85871:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85871:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85929:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85929:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85929:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85929:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85937:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85937:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85937:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85937:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85937:mlx:core:mlx_fabric_open():172 libfabric:85937:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85937:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85937:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85971:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85971:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85971:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85971:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85874:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85874:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85874:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85874:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85917:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85917:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85917:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85932:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85932:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85932:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85932:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85932:mlx:core:mlx_fabric_open():172 libfabric:85932:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85932:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85932:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85939:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85939:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85939:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85939:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85954:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85954:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85954:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85954:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85954:mlx:core:mlx_fabric_open():172 libfabric:85954:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85954:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85954:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85917:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85931:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85931:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85931:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85931:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85931:mlx:core:mlx_fabric_open():172 libfabric:85931:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85931:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85931:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85968:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85968:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85968:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85968:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85906:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85906:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85906:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85988:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85988:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85988:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85988:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85906:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85918:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85918:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85918:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85987:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85987:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85987:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85918:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85987:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85868:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85868:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85868:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85868:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85869:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85869:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85869:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85926:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85926:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85926:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85926:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85869:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85929:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85929:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85929:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85929:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85929:mlx:core:mlx_fabric_open():172 libfabric:85929:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85929:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85929:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85885:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85885:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85885:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85885:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85899:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85899:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85899:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85899:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85981:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85981:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85894:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85894:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85896:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85896:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85896:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85874:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85894:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85874:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85874:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85874:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85874:mlx:core:mlx_fabric_open():172 libfabric:85874:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85882:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85882:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85882:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85874:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85874:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85894:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85896:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85971:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85971:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85971:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85917:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85917:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85917:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85917:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85971:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85971:mlx:core:mlx_fabric_open():172 libfabric:85971:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85971:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85971:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85882:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85917:mlx:core:mlx_fabric_open():172 libfabric:85917:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85917:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85917:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85939:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85939:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85939:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85939:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85939:mlx:core:mlx_fabric_open():172 libfabric:85939:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85939:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85939:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85968:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85968:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85968:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85968:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85968:mlx:core:mlx_fabric_open():172 libfabric:85968:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85968:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85968:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85918:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85918:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85918:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85906:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85918:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85918:mlx:core:mlx_fabric_open():172 libfabric:85918:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85918:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85918:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85872:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85872:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85906:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85906:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85906:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85906:mlx:core:mlx_fabric_open():172 libfabric:85906:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85906:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85906:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85989:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85989:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85989:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85868:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85989:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85868:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85868:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85868:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85868:mlx:core:mlx_fabric_open():172 libfabric:85868:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85868:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85868:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85869:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85987:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85988:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85988:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85988:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85869:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85869:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85869:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85869:mlx:core:mlx_fabric_open():172 libfabric:85869:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85869:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85987:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85987:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85987:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85987:mlx:core:mlx_fabric_open():172 libfabric:85987:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85987:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85987:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85988:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85988:mlx:core:mlx_fabric_open():172 libfabric:85988:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85988:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85988:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85869:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85926:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85926:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85926:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85927:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85927:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85927:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85865:core:core:ofi_register_provider():474 registering provider: mlx (1.4) libfabric:85885:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85885:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85885:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85899:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85899:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85899:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85899:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85926:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85926:mlx:core:mlx_fabric_open():172 libfabric:85926:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85926:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85926:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85882:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85885:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85885:mlx:core:mlx_fabric_open():172 libfabric:85885:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85885:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85885:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85894:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85894:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85894:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85899:mlx:core:mlx_fabric_open():172 libfabric:85899:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85899:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85899:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85882:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85882:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85882:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85882:mlx:core:mlx_fabric_open():172 libfabric:85882:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85882:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85882:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85894:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85894:mlx:core:mlx_fabric_open():172 libfabric:85894:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85894:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85894:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85896:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85896:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85896:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85896:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85927:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85896:mlx:core:mlx_fabric_open():172 libfabric:85896:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85896:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85896:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85865:core:core:ofi_register_provider():474 registering provider: ofi_hook_noop (113.20) libfabric:85865:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85865:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85876:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85951:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85972:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85972:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85972:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85972:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85989:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85953:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85953:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85989:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85989:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85989:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85989:mlx:core:mlx_fabric_open():172 libfabric:85989:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85989:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85989:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85955:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85955:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85900:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85900:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85914:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85914:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85915:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85915:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85976:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85927:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85927:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85927:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85927:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85927:mlx:core:mlx_fabric_open():172 libfabric:85927:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85927:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85927:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85972:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85972:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85972:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85972:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85972:mlx:core:mlx_fabric_open():172 libfabric:85972:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85972:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85972:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85865:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85970:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85865:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85865:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85865:core:core:fi_getinfo_():1138 Found provider with the highest priority mlx, must_use_util_prov = 0 libfabric:85865:mlx:core:mlx_getinfo():211 primary detected device: mlx5_0 libfabric:85879:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85879:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85954:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85954:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85867:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85867:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85985:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85985:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85937:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85937:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85865:mlx:core:mlx_getinfo():254 used inject size = 1024 libfabric:85865:mlx:core:mlx_getinfo():301 Loaded MLX version 1.11.1 libfabric:85865:mlx:core:mlx_getinfo():348 MLX: spawn support 0 libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, psm3 has been skipped. To use psm3, please, set FI_PROVIDER=psm3 libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, verbs has been skipped. To use verbs, please, set FI_PROVIDER=verbs libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, tcp has been skipped. To use tcp, please, set FI_PROVIDER=tcp libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, sockets has been skipped. To use sockets, please, set FI_PROVIDER=sockets libfabric:85865:core:core:fi_getinfo_():1161 Since mlx can be used, shm has been skipped. To use shm, please, set FI_PROVIDER=shm libfabric:85865:mlx:core:mlx_fabric_open():172 libfabric:85865:core:core:fi_fabric_():1423 Opened fabric: mlx libfabric:85865:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85865:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85932:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85932:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85931:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85931:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85929:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85929:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85871:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85872:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85934:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85971:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85971:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85981:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85933:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85958:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85953:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85955:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85874:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85874:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85917:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85917:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85906:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85906:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85939:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85939:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85927:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85927:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85914:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85988:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85988:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85900:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85968:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85968:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85899:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85899:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85918:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85918:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85987:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85987:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85868:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85868:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85896:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85896:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85894:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85894:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85869:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85869:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85882:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85882:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85885:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85885:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85926:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85926:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85972:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85972:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85989:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85989:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85865:mlx:core:ofi_check_rx_attr():786 Tx only caps ignored in Rx caps libfabric:85865:mlx:core:ofi_check_tx_attr():884 Rx only caps ignored in Tx caps libfabric:85915:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85954:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85937:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85867:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85931:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85932:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85879:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85985:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85929:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85971:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85874:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85939:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85917:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85906:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85927:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85968:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85988:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85918:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85987:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85899:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85896:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85868:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85894:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85972:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85882:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85989:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85865:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85885:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85926:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:85869:mlx:core:mlx_cm_getname_mlx_format():73 Loaded UCP address: [307]... libfabric:17216:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xc3ad00 libfabric:17217:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x8fad60 libfabric:17218:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x159a2c0 libfabric:85865:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x13abf50 libfabric:17219:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1e83cd0 libfabric:85866:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x24bafd0 libfabric:17220:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x12c7cd0 libfabric:85867:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1583190 libfabric:17221:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xe22d20 libfabric:85868:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2a25190 libfabric:17222:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1a71cd0 libfabric:85869:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x18b1190 libfabric:17223:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xcbf930 libfabric:85870:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1a29fd0 libfabric:17224:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2724cd0 libfabric:85871:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b08190 libfabric:17225:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xb1bc00 libfabric:85872:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x20a9190 libfabric:17226:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xb4e3d0 libfabric:85873:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1432fd0 libfabric:17227:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xc74e60 libfabric:85874:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1ce4190 libfabric:17228:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x24c7b70 libfabric:85875:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xf3cfd0 libfabric:17229:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1444d60 libfabric:85876:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x21f7190 libfabric:17230:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2633c00 libfabric:85877:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1f30fd0 libfabric:17231:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x20122c0 libfabric:85878:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x14e2190 libfabric:17232:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x144eb00 libfabric:85879:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1e8c190 libfabric:17233:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xc9ce50 libfabric:85880:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1522fd0 libfabric:17234:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x24537c0 libfabric:85881:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x18f1fd0 libfabric:17235:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xb76b00 libfabric:85882:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xed8190 libfabric:17236:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x12737c0 libfabric:85883:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2882190 libfabric:17237:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xce0000 libfabric:85884:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1154190 libfabric:17238:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x12777c0 libfabric:85885:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x25b7190 libfabric:17239:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x7eb7c0 libfabric:85886:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x18b0fd0 libfabric:17240:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x11037c0 libfabric:85887:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xd77190 libfabric:17241:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x100a7c0 libfabric:85888:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1617fd0 libfabric:17242:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xcfab00 libfabric:85889:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xfa6fd0 libfabric:17243:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x216a7c0 libfabric:85890:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x10d6fd0 libfabric:17244:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1eecd00 libfabric:85891:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x247a190 libfabric:17245:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x17b2b90 libfabric:85892:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1f6e190 libfabric:17246:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xabac70 libfabric:85893:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b0bfd0 libfabric:17247:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1837be0 libfabric:85894:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x191f190 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17248:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xde1ce0 libfabric:85895:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xae9fd0 libfabric:17249:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x257d6d0 libfabric:85896:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x206a190 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17250:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x9eece0 libfabric:85897:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1502fd0 libfabric:17251:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x22eace0 libfabric:85898:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b33190 libfabric:17252:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1c058e0 libfabric:85899:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1f84190 libfabric:17253:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xd9a8e0 libfabric:85900:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x18ee190 libfabric:17254:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1e8a8c0 libfabric:85901:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xebafd0 libfabric:17255:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1bc58e0 libfabric:85902:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x26cefd0 libfabric:17256:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x146e8e0 libfabric:85903:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1660fd0 libfabric:17257:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x182dce0 libfabric:85904:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x22b5fd0 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17258:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x104b8e0 libfabric:85905:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b40190 libfabric:17259:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1f2e8e0 libfabric:85906:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1248190 libfabric:17260:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1d63880 libfabric:85907:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xdf5fd0 libfabric:17261:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1c746d0 libfabric:85908:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x12c1fd0 libfabric:17262:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x11c18e0 libfabric:85909:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x283bfd0 libfabric:17263:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x25ee8e0 libfabric:85910:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xdb8fd0 libfabric:17264:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1bbfb70 libfabric:85911:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x23e1fd0 libfabric:17265:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x281acd0 libfabric:85912:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1c60fd0 libfabric:17266:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xde8b70 libfabric:85913:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xf85fd0 libfabric:17267:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x17982c0 libfabric:85914:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1945190 libfabric:17268:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x21df2c0 libfabric:85915:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2862190 libfabric:17269:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x11f7b70 libfabric:85916:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2080fd0 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17270:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x19b2c10 libfabric:85917:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1d01190 libfabric:17271:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x164bd30 libfabric:85918:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x21dc190 libfabric:17272:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1360b70 libfabric:85919:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1207190 libfabric:17273:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x156ecd0 libfabric:85920:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2426fd0 libfabric:17274:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x219c2b0 libfabric:85921:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x18ca190 libfabric:17275:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xb2bb70 libfabric:85922:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xc47190 libfabric:17276:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x26fa2b0 libfabric:85923:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b02fd0 libfabric:17277:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1118ca0 libfabric:85924:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1730fd0 libfabric:17278:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1ceb2b0 libfabric:85925:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2997fd0 libfabric:17279:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x252d2b0 libfabric:85926:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x160a190 libfabric:17280:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x11698a0 libfabric:85927:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b93190 libfabric:17281:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xbf6360 libfabric:85928:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x180bfd0 libfabric:17282:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x19c4770 libfabric:85929:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x189a190 libfabric:17283:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b43360 libfabric:85930:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x171bfd0 libfabric:17284:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x17ad8a0 libfabric:85931:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xd37190 libfabric:17285:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x970c20 libfabric:85932:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1ad4190 libfabric:17286:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x7af360 libfabric:85933:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1408190 libfabric:17287:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xf70360 libfabric:85934:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b96190 libfabric:17288:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x20d5360 libfabric:85935:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2353fd0 libfabric:17289:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xfd28a0 libfabric:85936:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b6cfd0 libfabric:17290:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x250d360 libfabric:85937:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xfb7190 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17291:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x205a360 libfabric:85938:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x24e3fd0 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17292:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x136fc20 libfabric:85939:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x25a6190 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17293:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xe54e60 libfabric:85940:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x29e1fd0 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17294:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xe46d40 libfabric:85941:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1560fd0 libfabric:17295:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xca6360 libfabric:85942:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xedafd0 libfabric:17296:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x20a48b0 libfabric:85943:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xaf3190 libfabric:17297:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1008140 libfabric:85944:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x148bfd0 libfabric:17298:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x160f8b0 libfabric:85945:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x22e2fd0 libfabric:17299:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1d518a0 libfabric:85946:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1c45fd0 libfabric:17300:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x8fc140 libfabric:85947:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x100b190 libfabric:17301:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x263c8b0 libfabric:85948:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b32fd0 libfabric:17302:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x11fcc10 libfabric:85949:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1dd7fd0 libfabric:17303:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2537e70 libfabric:85950:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x27abfd0 libfabric:17304:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2626140 libfabric:85951:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x115c190 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17305:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2542c10 libfabric:85952:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2453fd0 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17306:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1522140 libfabric:85953:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x225e190 libfabric:17307:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x22c3180 libfabric:85954:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1377190 libfabric:17308:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x26b1180 libfabric:85955:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x242c190 libfabric:17309:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1bb68b0 libfabric:85956:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x290dfd0 libfabric:17310:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xc73140 libfabric:85957:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x262afd0 libfabric:17311:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x23318b0 libfabric:85958:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1f2f190 libfabric:17312:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x22452e0 libfabric:85959:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1675190 libfabric:17313:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x23e3b70 libfabric:85960:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1ac5fd0 libfabric:17314:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x11232e0 libfabric:85961:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2974fd0 libfabric:17315:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x14b0e40 libfabric:85962:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x243efd0 libfabric:17316:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x23551f0 libfabric:85963:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1944fd0 libfabric:17317:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1535c00 libfabric:85964:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1ff3190 libfabric:17318:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x17601f0 libfabric:85965:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x137afd0 libfabric:17319:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1bcacb0 libfabric:85966:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1296fd0 libfabric:17320:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x894e80 libfabric:85967:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x14f9fd0 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17321:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1421e80 libfabric:85968:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1716190 libfabric:17322:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x13f6b70 libfabric:85969:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2496190 libfabric:17323:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x14b2b90 libfabric:85970:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xbc5190 libfabric:17325:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xceccd0 libfabric:85971:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x202b190 libfabric:17326:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1c08d60 libfabric:85972:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1f30190 libfabric:17327:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1129cd0 libfabric:85973:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2651fd0 libfabric:17328:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x241e2e0 libfabric:85974:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1a21190 libfabric:17329:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x184ccb0 libfabric:85975:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1d72fd0 libfabric:17330:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xeefcb0 libfabric:85976:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x23b7190 libfabric:17331:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1ff2df0 libfabric:85977:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2359fd0 libfabric:17332:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1af1b00 libfabric:85978:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xbb6190 libfabric:17333:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1c392d0 libfabric:85979:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xc96190 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17334:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xfb5af0 libfabric:85980:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b52190 libfabric:17335:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1d78af0 libfabric:85981:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1022190 libfabric:17336:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1b42cd0 libfabric:85982:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2311fd0 libfabric:17337:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1cb6b00 libfabric:85983:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1f18190 libfabric:17338:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1079b90 libfabric:85984:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1821fd0 libfabric:17339:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x169dcd0 libfabric:85985:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1381190 libfabric:17340:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1f25af0 libfabric:85986:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x10e7fd0 libfabric:17341:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x236e2d0 libfabric:85987:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1e01190 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17342:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x111a3d0 libfabric:85988:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1c34190 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed libfabric:17343:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1738a90 libfabric:85989:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1037190 libfabric:17344:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x1502af0 libfabric:85990:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0xf16fd0 libfabric:85991:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x12f3fd0 libfabric:85992:mlx:core:mlx_av_insert():179 Try to insert address #0, offset=0 (size=2) fi_addr=0x2963fd0 Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143)........: MPID_Init(1310)..............: MPIDI_OFI_mpi_init_hook(1974): OFI get address vector map failed