topic Re: How to set the size of CQ? in Intel® MPI Library

How to set the size of CQ?

oleotiger — Fri, 03 Sep 2021 06:19:04 GMT

From the device information with command ibv_devinfo -v, I got

        max_cq:                         20480
        max_cqe:                        65535

The max number cqe is 65535.

With openmpi, I can set UCX_RC_TX_CQ_LEN(default 4096) to 65535. This value can gurantee that the length of CQE will not exceed the limit of configuration of the network card.

Is there envrionment variable of libfabric or mpi that can limit the number of CQE?

Re:How to set the size of CQ?

VarshaS_Intel — Mon, 06 Sep 2021 11:53:59 GMT

Hi,

Thanks for reaching out to us.

We are working on your issue and we will get back to you soon.

Thanks & Regards

Varsha

Re: How to set the size of CQ?

VarshaS_Intel — Thu, 09 Sep 2021 06:59:29 GMT

Hi,

Could you please specify to us which libfabric provider(Mellanox / PSM2 / Verbs) you are using currently?

Thanks & Regards

Varsha

Re: How to set the size of CQ?

oleotiger — Fri, 10 Sep 2021 06:53:31 GMT

I have two servers with differernt libfabric providers.

One is equipped with Mellanox network card ( RoCE protocal ).

One is equipped with Huawei Hi1822 network card ( RoCE supported ).

The max number of CQE of Hi1822 is 65535. So I;m looking for a way to set the max number of CQE of transport layer.

OpenUCX can control the number of CQR with environment variables.

Is there any way of libfabric or intelmpi that can limit the max number of cqe created?

Re:How to set the size of CQ?

VarshaS_Intel — Mon, 13 Sep 2021 08:51:37 GMT

Hi,

Thank you for providing the information.

>>>With openmpi, I can set UCX_RC_TX_CQ_LEN(default 4096) to 65535.

Could you please let us know how you are checking that OpenMPI is obeying the setting of UCX_RC_TX_CQ_LEN?

Thanks & Regards

Varsha

Re: Re:How to set the size of CQ?

oleotiger — Thu, 16 Sep 2021 12:41:13 GMT

As the infomation of the network card shown above, the max_cqe of my network card is 65535.

With setting UCX_RC_TX_CQ_LEN to 65535 or any number that is equal or less than 65535, I can run OSU benchmark with it.

Once setting UCX_RC_TX_CQ_LEN to 65536 or any number that is large than 65535, I can run OSU. It will raise exceptions.

So I get the conclusion that OpenMPI is obeying the setting of UCX_RC_TX_CQ_LEN.

Re: How to set the size of CQ?

VarshaS_Intel — Thu, 23 Sep 2021 12:26:12 GMT

Hi,

Thanks for providing all the requested information.

We want to add some points regarding the UCX:

•The Intel MPI uses UCX in the backend for Infiniband. The UCX variables are specific to the UCX framework and are not specific to Intel MPI/OpenMPI.

•UCX is a collaboration between industry, laboratories and academia to create an open-source production grade communication framework for data centric and HPC applications.

>>> Is there any way of libfabric or intelmpi that can limit the max number of cqe created?

As long as UCX obeys the setting of UCX_RC_TX_CQ_LEN, the MPI application is expected to see the effects of setting this UCX variable UCX_RC_TX_CQ_LEN.

Inline with your experiment, we have followed all the steps provided by you at our end with Intel MPI and we found a similar behavior which you got with OpenMPI.

When we run the command "ibv devinfo -v" we got the max_cqe = 4194303.

Also, we want to mention that we can not say that certain MPI library is obeying/disobeying the setting UCX_RC_TX_CQ_LEN.

Thanks & Regards

Varsha

Re: How to set the size of CQ?

oleotiger — Thu, 30 Sep 2021 01:04:18 GMT

If I'm using OFED, IntelMPI is using the stack : Libfabric-->OFED-->IB verbs/api. In this way there is no UCX. I want to know that if application is running in this way, is there any way that we can control the size of CQE ( as what UCX_RC_TX_CQ_LEN does)?

Re: How to set the size of CQ?

VarshaS_Intel — Mon, 04 Oct 2021 11:46:27 GMT

Hi,

>>>I want to know that if application is running in this way, is there any way that we can control the size of CQE ( as what UCX_RC_TX_CQ_LEN does)?

UCX library will be used by Intel MPI Library only when the mlx provider is used. On the verbs side, such a control (i.e. UCX_RC_TX_CQ_LEN) does not exist.

Thanks & Regards

Varsha

Re:How to set the size of CQ?

VarshaS_Intel — Tue, 12 Oct 2021 05:27:49 GMT

Hi,

We haven't heard back from you. Could you please provide an update?

Thanks & Regards

Varsha

Re:How to set the size of CQ?

VarshaS_Intel — Wed, 20 Oct 2021 04:52:16 GMT

Hi,

We have not heard back from you. This thread will no longer be monitored by Intel. If you need further assistance, please post a new question.

Thanks & Regards

Varsha