OpenMP Globally Limit Threads

kbsart · ‎03-14-2023

Hello,

We are using OMP for some of our applications parallelism. We have the desire to limit the number of OMP threads. We are aware of setting environmental variables (e.g. OMP_NUM_THREADS=1), however this will not work for our usecase as we only wish certain processes to still be able to have multiple threads.

The "omp_set_num_threads" API does not also seem to work in the way we want it to. For example we have multiple test cases similar to the following:

	TEST(IppMklOmpThreadControlTests, OmpSetNumThreads_EnvVariable_LimitsOmpChildThreads)
	{
		unique_lock<mutex> lck{ oneAtATime };

		auto threadStarted = ManualResetEvent::Create();
		auto threadNumChanged = ManualResetEvent::Create();

		int setNumThreads = 1;
		int parentNumThreads = 0;
		int childNumThreads = 0;

		char numThreadChildValue[2] = "x";
		char threadLimitChildValue[2] = "x";
		char dynamicSettingChildValue[2] = "x";
		size_t envVarSize = 2;

		// Set OMP env variables
		_putenv_s("OMP_NUM_THREADS", to_string(1).c_str());
		_putenv_s("OMP_DYNAMIC", to_string(0).c_str());
		_putenv_s("OMP_THREAD_LIMIT", to_string(1).c_str());

		// OMP thread API setters
		omp_set_dynamic(0);
		omp_set_num_threads(setNumThreads);

		thread th = thread{ [&]()
			{
				childNumThreads = omp_get_max_threads();

				// Get child env variables
				getenv_s(&envVarSize, numThreadChildValue, envVarSize, "OMP_NUM_THREADS");
				getenv_s(&envVarSize, threadLimitChildValue, envVarSize, "OMP_DYNAMIC");
				getenv_s(&envVarSize, dynamicSettingChildValue, envVarSize, "OMP_THREAD_LIMIT");
			} };

		th.join();

		EXPECT_EQ(setNumThreads, childNumThreads);
	}

If we run this test function by itself, the number of child threads is as we expect. However if multiple test cases are run sequentially, this test case fails (unless of course we set env variables).

Is there a way to for all child threads to use a global OMP thread limit without setting environment variables?

Thanks in advance for your support

PraneethA_Intel · ‎03-15-2023

Hi Kevin,

Thanks for posting in Intel communities.

Could you please get back to us with the following details so that we could look into your issue.

1. Sample reproducer of the code

2. Steps to reproduce (if any)

3. Environment details (Operating System etc)

Thanks and Regards,

Praneeth Achanta

kbsart · ‎03-15-2023

Hi Praneeth,

Thanks for taking a look at this!

We are using VS 2022 as our IDE / build tool, and specifically using the TestExplorer feature using googletest to test the code. The following is a CPP file that reproduces the behavior on my end:

#include "pch.h"


#include "omp.h"
#include "mkl.h"
#include "ipp.h"
#include <thread>
#include <mutex>


using namespace std;

namespace EmbedTest
{
	namespace
	{
		static std::mutex oneAtATime;
	}
	TEST(_SharedIntelTests, MklSetNumThreads_WhenSetFromParentThread_IsInheritedByChildThreads)
	{
		unique_lock<mutex> lck{ oneAtATime };

		int setNumThreads = 1;
		int parentNumThreads = 0;
		int childNumThreads = 0;

		mkl_set_dynamic(0);
		mkl_set_num_threads(setNumThreads);
		parentNumThreads = mkl_get_max_threads();

		thread th = thread{ [&]()
			{
				childNumThreads = mkl_get_max_threads();
			} };


		th.join();

		EXPECT_EQ(setNumThreads, parentNumThreads);
		EXPECT_EQ(parentNumThreads, childNumThreads);
	}

	TEST(_SharedIntelTests, OmpSetNumThreads_WhenSetFromParentThread_IsNotInheritedByChildThreads)
	{
		unique_lock<mutex> lck{ oneAtATime };

		int setNumThreads = 12;
		int parentNumThreads = 0;
		int childNumThreads = 0;

		omp_set_dynamic(0);
		omp_set_num_threads(setNumThreads);
		parentNumThreads = omp_get_max_threads();

		thread th = thread{ [&]()
			{
				childNumThreads = omp_get_max_threads();
			} };


		th.join();

		EXPECT_EQ(setNumThreads, parentNumThreads);
		EXPECT_NE(parentNumThreads, childNumThreads);
	}

	TEST(_SharedIntelTests, OmpSetNumThreads_EnvVariable_LimitsOmpChildThreads)
	{
		unique_lock<mutex> lck{ oneAtATime };

		int setNumThreads = 1;
		int parentNumThreads = 0;
		int childNumThreads = 0;

		char numThreadChildValue[2] = "x";
		char threadLimitChildValue[2] = "x";
		char dynamicSettingChildValue[2] = "x";
		size_t envVarSize = 2;

		// Set OMP env variables
		_putenv_s("OMP_NUM_THREADS", to_string(1).c_str());
		_putenv_s("OMP_DYNAMIC", to_string(0).c_str());
		_putenv_s("OMP_THREAD_LIMIT", to_string(1).c_str());

		// OMP thread API setters
		omp_set_dynamic(0);
		omp_set_num_threads(setNumThreads);

		thread th = thread{ [&]()
			{
				childNumThreads = omp_get_max_threads();

				// Get child env variables
				getenv_s(&envVarSize, numThreadChildValue, envVarSize, "OMP_NUM_THREADS");
				getenv_s(&envVarSize, threadLimitChildValue, envVarSize, "OMP_DYNAMIC");
				getenv_s(&envVarSize, dynamicSettingChildValue, envVarSize, "OMP_THREAD_LIMIT");
			} };

		th.join();

		EXPECT_EQ(setNumThreads, childNumThreads);
	}

}

To reproduce the error, I run all these _SharedIntelTests inside the VS2022 test explorer, e.g.

Specifically we wouldn't expect the "OmpSetNumThreads_EnvVariable_LimitsOmpChildThreads" to fail.

As an interesting aside, if I only run the "OmpSetNumThreads_EnvVariable_LimitsOmpChildThreads" test by itself, the test passes. It fails when running all the tests together though. Passing test screenshot using same code:

For environment, as mentioned we are using VS2022 with Windows 10 Enterprise Version 21H2 (OS build 19044.2604). The Libiomp version we are using appears to be 2021.4.0

Thanks again for reaching out

PraneethA_Intel · ‎03-23-2023

Hi Kevin,

Thanks for providing the details.

We have tried the below steps at our end to run the code and have noticed that the "OmpSetNumThreads_EnvVariable_LimitsOmpChildThreads" did not fail even while running all the tests at the same time.(See below screenshot)

1. Create new google test project in Visual Studio

2. Added your code in the test.cpp file

3. Run all test in the Test Explorer(Under Test -> Test Explorer)

Please follow the above steps and check if your issue is resolved, if it isn't could you please provide us with the exact steps you have used when you encountered this issue.

Thanks and Regards,

Praneeth Achanta

PraneethA_Intel · ‎03-29-2023

Hi Kevin,

We have not heard back from you. Could you please provide an update on your issue?

Thanks & Regards,

Praneeth Achanta

PraneethA_Intel · ‎04-05-2023

Hi Kevin,

We have not heard back from you. This thread will no longer be monitored by Intel. If you need further assistance, please post a new question.

Thanks and Regards,

Praneeth Achanta