topic Re: 'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non- in Intel® oneAPI Math Kernel Library

'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-pow 2

ahnitz — Thu, 15 Jun 2023 21:33:42 GMT

We are trying to track down an issue where we are unable to use the MKL FFT library to do certain sizes of 1d transforms. From experimentation it seems that any size is allowed if it less than 2^24 in length, but only powers of 2 are allowed past that point (so 2^25, 26, etc).

The documentation would seem to indicate that arbitrary sizes are allowed up to a much higher limit than this, but perhaps there is a issue with how we are setting up the descriptor. Does anyone have suggestions for what our mistake may have been, or can anyone confirm that they have some example of a 1d FFT working for non-power-of-two sizes beyond 2^24?

We are running through python, so below is a paired down example that pulls in the mkl runtime library using ctypes. This should be reproduceable using the version of mkl on pypi (2013.1.0).

import os, sys, ctypes, numpy, fnmatch # pick up lib from pypi mkl wheel libdir = os.path.join(sys.prefix, "lib") libs = [fn for fn in os.listdir(libdir) if fnmatch.fnmatch(fn, 'libmkl_rt.so*')] lib = ctypes.CDLL(os.path.join(libdir, libs[0])) #MKL constants taken from mkl_df_defines.h DFTI_REAL = 33 DFTI_DOUBLE = 36 def check_status(status): if status: lib.DftiErrorMessage.restype = ctypes.c_char_p msg = lib.DftiErrorMessage(status) raise RuntimeError(msg) def fft(N): invec = numpy.zeros(N, dtype=numpy.float64) # Create input / output outvec = numpy.zeros(N//2 + 1, dtype=numpy.complex128) desc = ctypes.c_void_p(1) # Setup descriptor f = lib.DftiCreateDescriptor f.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_int, ctypes.c_int] status = f(ctypes.byref(desc), DFTI_DOUBLE, DFTI_REAL, 1, N) lib.DftiCommitDescriptor(desc) check_status(status) f = lib.DftiComputeForward # Run fft f.argtypes = [ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p] status = f(desc, invec.ctypes.data, outvec.ctypes.data) check_status(status) for N in [2**24, 2**25, 2**24+100]: try: fft(int(N)) print(f'{N} works') except RuntimeError as e: print(f'{N} fails') raise e

The output is the following

16777216 works 33554432 works 16777316 fails Traceback (most recent call last): File "/home/ahnitz/projects/debug/m2.py", line 40, in <module> raise e File "/home/ahnitz/projects/debug/m2.py", line 36, in <module> fft(int(N)) File "/home/ahnitz/projects/debug/m2.py", line 32, in fft check_status(status) File "/home/ahnitz/projects/debug/m2.py", line 16, in check_status raise RuntimeError(msg) RuntimeError: b'Intel MKL DFTI ERROR: Invalid configuration parameters'

Re:'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-pow 2

ShanmukhS_Intel — Mon, 19 Jun 2023 14:30:26 GMT

Hi Alexander,

Thanks for posting in Intel communities.

Thanks for sharing the details. We have tried reproducing the shared code at our end. Please find the results below. we will get back to you soon with an update.

16777216 works

33554432 works

16777316 fails

Traceback (most recent call last):

File "/home/abc/samply.py", line 40, in <module>

raise e

File "/home/abc/samply.py", line 36, in <module>

fft(int(N))

File "/home/abc/samply.py", line 32, in fft

check_status(status)

File "/home/abc/samply.py", line 16, in check_status

raise RuntimeError(msg)

RuntimeError: b'Intel MKL DFTI ERROR: Invalid configuration parameters'

Best Regards,

Shanmukh.SS

Re: Re:'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and n

ahnitz — Thu, 22 Jun 2023 17:58:39 GMT

@ShanmukhS_Intel thanks for verifying this problem. Can you confirm if the behavior is intended or not? The documentation seems to indicate that while ipp has some limitations that MKL is generic in regard to transform size.

https://www.intel.com/content/www/us/en/developer/articles/technical/onemkl-ipp-choosing-an-fft.html

Re: 'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-

ShanmukhS_Intel — Thu, 29 Jun 2023 12:00:28 GMT

Hi Alexander,

Seems the behavior is not an intended one as per the below data. We are discussing this issue internally. We will be back with an update soon.

	MKL	IPP
Transform Sizes	32-bit platforms - maximum size is 2^31-1	FFT - Powers of 2 only (**)
Transform Sizes	64-bit platforms - 2⁶⁴ maximum size	DFT -2³² maximum size (**)

Best Regards,

Shanmukh.SS

Re:'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-pow 2

ShanmukhS_Intel — Thu, 06 Jul 2023 17:25:47 GMT

Hi Alexander,

This should be reproduceable using the version of mkl on pypi (2013.1.0).

>> Seems the issue is reproducible with 2023.1.0 and considering the above as a typo. We would like you to correct us if we are wrong.

Best Regards,

Shanmukh.SS

Re: Re:'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and n

ahnitz — Thu, 06 Jul 2023 17:29:13 GMT

Thanks for the catch, yes that was a typo! I did intend to say 2023.1.0 as you already surmised.

Re: 'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-

ShanmukhS_Intel — Wed, 26 Jul 2023 14:27:19 GMT

Hi Alexander,

Thanks for the confirmation. There was some discrepancy in the sizes being mentioned.

We are working on your issue internally. We will get back to you soon with an update regarding the size confirmation.

Best Regards,

Shanmukh.SS

Re:'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-pow 2

ShanmukhS_Intel — Wed, 26 Jul 2023 14:40:05 GMT

Hi Alexander,

We have tested all the sizes mentioned earlier by you, by running the code sample provided by Intel MKL. We could see the size 16777316 worked fine, which failed in the previous case as you mentioned. However, we could see some inconsistencies with certain other valid sizes, and we are investigating and working on the resolution. Thanks for letting us know. Below is the log for your reference.

Intel(R) oneAPI Math Kernel Library Version 2023.2-Product Build 20230613 for Intel(R) 64 architecture applications

Example basic_sp_real_dft_1d

Forward-Backward single-precision real-to-complex out-of-place 1D transform

Configuration parameters:

DFTI_PRECISION = DFTI_SINGLE

DFTI_FORWARD_DOMAIN = DFTI_REAL

DFTI_DIMENSION = 1

DFTI_LENGTHS = {16777316}

DFTI_PLACEMENT = DFTI_NOT_INPLACE

DFTI_CONJUGATE_EVEN_STORAGE = DFTI_COMPLEX_COMPLEX

Create DFTI descriptor

Set configuration: out-of-place

Set configuration: CCE storage

Commit the descriptor

Allocate data arrays

Initialize data for real-to-complex FFT

Compute forward transform

Verify the result

Check if err is below errthr 7.15e-06

Verified, maximum error was 2.92e-07

Initialize data for complex-to-real FFT

Compute backward transform

Verify the result

Check if err is below errthr 7.15e-06

Verified, maximum error was 2.38e-07

Free DFTI descriptor

Free data arrays

TEST PASSED

In addition, you could refer to the sample provided by Intel MKL under the below path for reference.

C:\Program Files (x86)\Intel\oneAPI\mkl\2023.2.0\examples\examples_core_c.zip\c\dft\source\basic_sp_real_dft_1d.c

Best Regards,

Shanmukh.SS

Re: Re:'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and n

ahnitz — Thu, 17 Aug 2023 17:38:47 GMT

Is there any update on this? Does it seem sensible that whatever issue was identified would explain why this doesn't work with the python distributed versions rather than directly compiling this version?

Re: 'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-

ahnitz — Wed, 13 Sep 2023 02:39:25 GMT

Is there any update on this issue? Is a resolution still being worked out or is there any information to clarify what the problem is?

Re: 'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-

ahnitz — Fri, 20 Oct 2023 19:27:45 GMT

This issue is still unresolved in the latest versions. e.g. mkl-2023.2.0-py2.py3-none-manylinux1_x86_64.whl, the same test script posted above still fails. Is there any progress on the resolution?

Re:'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-pow 2

ShanmukhS_Intel — Mon, 04 Dec 2023 19:21:16 GMT

Hi Alexander,

The reason the Python calls behave differently from those in C is that they are NOT the same calls. DftiCreateDescriptor is a preprocessor macro (as seen in

mkl_dfti.h) that may be "resolved" to any of 4 special cases ({single,double}x{1d,md}), before falling back to the original library "symbol" DftiCreateDescriptor.

For the double-precision, 1D case here, DftiCreateDescriptor is "resolved" by the preprocessor to DftiCreateDescriptor_d_1d (with a different signature), which

"resolution" does NOT happen with Python's ctypes.CDLL('libmkl_rt.so').DftiCreateDescriptor.

Hence, a different (unsupported) code path is taken in Python, whereas in C

there is no issue in this case.

For this and other intricacies (including possible type mismatches), extracting the symbols from the library (e.g., libmkl_rt.so) is NOT recommended or supported.

However, if the you still wish to go down this unsupported, unrecommended path, you can use

f = lib.DftiCreateDescriptor_d_1d

f.argtypes = [ctypes.POINTER(ctypes.c_void_p), ctypes.c_int,

ctypes.c_long]

status = f(ctypes.byref(desc), DFTI_REAL, N)

instead of the

f = lib.DftiCreateDescriptor

f.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_int, ctypes.c_int]

status = f(ctypes.byref(desc), DFTI_DOUBLE, DFTI_REAL, 1, N)

Regrets for the delay in the response.

Best Regards,

Shanmukh.SS

Re: 'Intel MKL DFTI ERROR: Invalid configuration parameters' for 1d FFT with size > 2^24 and non-

ShanmukhS_Intel — Fri, 22 Dec 2023 08:54:11 GMT

Hi Alexander,

A gentle reminder:

Has the information provided earlier helped? Could you please get back to us if you have any update on your issue or you could raise a new thread if you have any issues.

Best Regards,

Shanmukh.SS