oneCCL: what is the purpose of key value store in create_communicator?

[sogartar]

I don't understand what is the `kvs` parameter in create_communicator.

In this example I see it being used like that

    /* create kvs */
    ccl::shared_ptr_class<ccl::kvs> kvs;
    ccl::kvs::address_type main_addr;
    if (rank == 0) {
        kvs = ccl::create_main_kvs();
        main_addr = kvs->get_address();
        MPI_Bcast((void*)main_addr.data(), main_addr.size(), MPI_BYTE, 0, MPI_COMM_WORLD);
    }
    else {
        MPI_Bcast((void*)main_addr.data(), main_addr.size(), MPI_BYTE, 0, MPI_COMM_WORLD);
        kvs = ccl::create_kvs(main_addr);
    }

    /* create communicator */
    auto dev = ccl::create_device(q.get_device());
    auto ctx = ccl::create_context(q.get_context());
    auto comm = ccl::create_communicator(size, rank, dev, ctx, kvs);

What are valid values for it?

[SantoshY_Intel]

Hi,

Thanks for posting in the Intel communities.

kvs_interface defines the key-value store (KVS) interface to be used to establish a connection between ranks during the creation of oneCCL communicator. For more information, please refer to the below link:

https://spec.oneapi.io/versions/1.0-rev-3/elements/oneCCL/source/spec/main_objects.html#key-value-store

Thanks & Regards,

Santosh

[sogartar]

Thanks, but the documentation is not helpful.

[SantoshY_Intel]

Hi,

>>>"I don't understand what is the `kvs` parameter in create_communicator."

The kvs parameter in create_communicator() expects an object of shared_ptr_class<kvs>(key-value store).

kvs_interface defines the key-value store (KVS) interface to be used to establish a connection between ranks during the creation of oneCCL communicator.

To create KVS objects before passing them to create_communicator(), use the below steps:

oneCCL specification defines kvs class as a built-in KVS provided by oneCCL.

1. Retrieve an address of built-in key-value store using the below function: const addr_t& kvs::get_addr() const; which returns kvs::addr_t

The address of the key-value store should be retrieved from the main built-in KVS and distributed to other processes for the built-in KVS creation.

Creating a main built-in key-value store:

Its address should be distributed using an out-of-band communication mechanism and be used to create key-value stores on other ranks using the function: kvs_t environment::create_main_kvs() const; which returns the main key-value store object kvs_t

Example:

ccl::shared_ptr_class<ccl::kvs> kvs;
  ccl::kvs::address_type main_addr;
  if (rank == 0) {
    kvs = ccl::create_main_kvs();
    main_addr = kvs->get_address();
    MPI_Bcast((void*)main_addr.data(), main_addr.size(), MPI_BYTE, 0, MPI_COMM_WORLD);
  }

2. Creating a new key-value store from main kvs address using the function: kvs_t environment::create_kvs(const kvs::addr_t& addr) const;

where, addr is the address of the main kvs & the above function returns kvs_t (key-value store object).

Example:

kvs = ccl::create_kvs(main_addr);

>>>"What are valid values for it?"

We should pass an object instance of shared_ptr_class<kvs>(key-value store) as a value.

Example: ccl::create_communicator(size, rank, dev, ctx, kvs);

Thanks & Regards,

Santosh

[SantoshY_Intel]

Hi,

We haven't heard back from you. Could you please provide us with an update on your issue?

Thanks & Regards,

Santosh

[SantoshY_Intel]

Hi,

I assume that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Thanks & Regards,

Santosh