About WideCharToMultiByte (Win32 API)

[Mincheol_L_]

Hello.

I have converted the MBCS(ANSI/UTF8) to Widecharacter using MultiByteToWideChar successfully.
But when I convert it back to MBCS(ANSI/UTF8) using WideCharToMultiByte, it always return zero(fails).
I am using Intel(R) Visual FORTRAN Compiler XE 12.1.7.371 [IA-32] and Intel(R) Visual FORTRAN Compiler XE 14.0.4.237 [IA-32]. 
The results are the same.

Please help me.

[Steven_L_Intel1]

Can you provide us with a small test case?

[Mincheol_L_]

Hello Steve

I made a small test case.
Please find the attached file.
Thanks a lot.

** note **
acp.f90 -> saved as US-ASCII (Codepage 20127)
utf8.f90 -> saved as Unicode (UTF8 with signature) (Codepage 65001)

[JVanB]

I tried composing an example in gfortran:

module ifwina
   use ISO_C_BINDING
   implicit none
   integer, parameter :: UINT = C_INT, DWORD = C_LONG, WCHAR = C_INT16_T
   integer, parameter :: BOOL = C_INT
   integer(UINT), parameter :: CP_ACP = 0
   integer(UINT), parameter :: CP_MACCP = 2
   integer(UINT), parameter :: CP_OEMCP = 1
   integer(UINT), parameter :: CP_SYMBOL = 42
   integer(UINT), parameter :: CP_THREAD_ACP = 3
   integer(UINT), parameter :: CP_UTF7 = 65000
   integer(UINT), parameter :: CP_UTF8 = 65001
   integer(DWORD), parameter :: MB_COMPOSITE = 2
   integer(DWORD), parameter :: MB_ERR_INVALID_CHARS = 8
   integer(DWORD), parameter :: MB_PRECOMPOSED = 1
   integer(DWORD), parameter :: MB_USEGLYPHCHARS = 4
   integer(DWORD), parameter :: WC_COMPOSITECHECK = 512
   integer(DWORD), parameter :: WC_ERR_INVALID_CHARS = 128
   integer(DWORD), parameter :: WC_NO_BEST_FIT_CHARS = 1024
   interface
      function MultiByteToWideChar(CodePage,dwFlags,lpMultiByteStr, &
         cbMultiByte,lpWideCharStr,cchWideChar) bind(C,name='MultiByteToWideChar')
         import
         implicit none
         !GCC$ ATTRIBUTES STDCALL :: MultiByteToWideChar
         !DEC$ ATTRIBUTES STDCALL :: MultiByteToWideChar
         integer(C_INT) MultiByteToWideChar
         integer(UINT), value :: CodePage
         integer(DWORD), value :: dwFlags
         character(KIND=C_CHAR) lpMultiByteStr(*)
         integer(C_INT), value :: cbMultiByte
         integer(WCHAR) lpWideCharStr(*)
         integer(C_INT), value :: cchWideChar
      end function MultiByteToWideChar
   end interface
   interface
      function WideCharToMultiByte(CodePage,dwFlags,lpWideCharStr, &
         cchWideChar,lpMultiByteStr,cbMultiByte,lpDefaultChar, &
         lpUsedDefaultChar) bind(C,name='WideCharToMultiByte')
         import
         implicit none
         !GCC$ ATTRIBUTES STDCALL :: WideCharToMultiByte
         !DEC$ ATTRIBUTES STDCALL :: WideCharToMultiByte
         integer(C_INT) WideCharToMultiByte
         integer(UINT), value :: CodePage
         integer(DWORD), value :: dwFlags
         integer(WCHAR) lpWideCharStr(*)
         integer(C_INT), value :: cchWideChar
         character(KIND=C_CHAR) lpMultiByteStr(*)
         integer(C_INT), value :: cbMultiByte
         character(KIND=C_CHAR) lpDefaultChar
         integer(BOOL) lpUsedDefaultChar
      end function WideCharToMultiByte
   end interface
   interface
      function GetLastError() bind(C,name='GetLastError')
         import
         implicit none
         !GCC$ ATTRIBUTES STDCALL :: GetLastError
         !DEC$ ATTRIBUTES STDCALL :: GetLastError
         integer(DWORD) GetLastError
      end function GetLastError
   end interface
end module ifwina

program mbcs
   use ifwina
   use ISO_C_BINDING
   implicit none
   integer(C_INT) iret1
   integer(DWORD) iret2
   character(LEN=256,KIND=C_CHAR) lpMultiByteStr
   integer(WCHAR) lpWideCharStr(LEN(lpMultiByteStr))
   integer(UINT) CodePage
   integer(DWORD) dwFlags
   integer(C_INT) cbMultiByte
   integer(C_INT) cchWideChar
   character(KIND=C_CHAR), pointer :: lpDefaultChar
   integer(BOOL), pointer :: lpUsedDefaultChar

   CodePage = CP_UTF8
   dwFlags = MB_ERR_INVALID_CHARS
   lpMultiByteStr = 'Hello world'//ACHAR(0)
   cbMultiByte = -1
   lpWideCharStr = lpWideCharStr
   cchWideChar = size(lpWideCharStr)
   iret1 = MultiByteToWideChar(CodePage,dwFlags, &
      lpMultiByteStr,cbMultiByte,lpWideCharStr,cchWideChar)
   if(iret1 == 0) then
      iret2 = GetLastError()
      write(*,'(a,i0)') 'MultiByteToWideChar failed with error code ',iret2
   else
      write(*,'(a,i0,a)') 'MultiByteToWideChar converted ',iret1,' characters'
   end if
   lpMultiByteStr = ''
   dwFlags = WC_ERR_INVALID_CHARS
   cbMultiByte = len(lpMultiByteStr)
   call C_F_POINTER(C_NULL_PTR,lpDefaultChar)
   call C_F_POINTER(C_NULL_PTR,lpUsedDefaultChar)
   cchWideChar = -1
   iret1 = WideCharToMultiByte(CodePage,dwFlags,lpWideCharStr,cchWideChar, &
      lpMultiByteStr,cbMultiByte,lpDefaultChar,lpUsedDefaultChar)
   if(iret1 == 0) then
      iret2 = GetLastError()
      write(*,'(a,i0)') 'WideCharToMultiByte failed with error code ',iret2
   else
      write(*,'(a,i0,a)') 'MultiByteToWideChar converted ',iret1,' characters'
      write(*,'(a)') 'lpMultiByteStr = ',lpMultiByteStr(1:index(lpMultiByteStr,ACHAR(0))-1)
   end if
end program mbcs

It worked, so I compared with your example and noticed that you passed a NULL pointer to the lpMultiByteStr argument of WideCharToMultiByte. When you get problems like this, you should check the error code returned by GetLastError which should be 122 for ERROR_INSUFFICIENT_BUFFER in this case.

[JVanB]

It looks like I was a little too hasty in my previous remark. lpMultiByteStr can be NULL if cbMultiByte is zero, which it was in your example. I have modified my example to do this and also to use a MesssageBoxW to print out the WCHAR string. I still don't get any errors. What does GetLastError return after the call to WideCharToMultiByte? Does my example work on your machine? Can you copy ifort's interface block (from kernel32.f90); maybe there is some problem with it. OK, looking at an old version of ifort's interface block, I see a problem: ifort doesn't include the ALLOW_NULL attribute for the lpUsedDefaultChar argument, so if this is still the case with current ifort, the compiler will send a non-NULL address for lpUsedDefaultChar and GetLastError should be returning 87 for ERROR_INVALID_PARAMETER. You can work around this by setting a pointer to point at what C_NULL_PTR points at and passing that as in the usage of lpUsedDefaultChar in my example, expanded below:

module ifwina
   use ISO_C_BINDING
   implicit none
   integer, parameter :: UINT = C_INT, DWORD = C_LONG, WCHAR = C_INT16_T
   integer, parameter :: BOOL = C_INT, HANDLE = C_INTPTR_T
   integer(UINT), parameter :: CP_ACP = 0
   integer(UINT), parameter :: CP_MACCP = 2
   integer(UINT), parameter :: CP_OEMCP = 1
   integer(UINT), parameter :: CP_SYMBOL = 42
   integer(UINT), parameter :: CP_THREAD_ACP = 3
   integer(UINT), parameter :: CP_UTF7 = 65000
   integer(UINT), parameter :: CP_UTF8 = 65001
   integer(DWORD), parameter :: MB_COMPOSITE = 2
   integer(DWORD), parameter :: MB_ERR_INVALID_CHARS = 8
   integer(DWORD), parameter :: MB_PRECOMPOSED = 1
   integer(DWORD), parameter :: MB_USEGLYPHCHARS = 4
   integer(DWORD), parameter :: WC_COMPOSITECHECK = 512
   integer(DWORD), parameter :: WC_ERR_INVALID_CHARS = 128
   integer(DWORD), parameter :: WC_NO_BEST_FIT_CHARS = 1024
   integer(DWORD), parameter :: WC_DEFAULTCHAR = 64
   integer(DWORD), parameter :: WC_DISCARDNS = 16
   integer(DWORD), parameter :: WC_SEPCHARS = 32
   integer(C_INT), parameter :: MB_ABORTRETRYIGNORE = 2
   integer(C_INT), parameter :: MB_CANCELTRYCONTINUE = 6
   integer(C_INT), parameter :: MB_HELP = 16384
   integer(C_INT), parameter :: MB_OK = 0
   integer(C_INT), parameter :: MB_OKCANCEL = 1
   integer(C_INT), parameter :: MB_RETRYCANCEL = 5
   integer(C_INT), parameter :: MB_YESNO = 4
   integer(C_INT), parameter :: MB_YESNOCANCEL = 3
   integer(C_INT), parameter :: MB_ICONEXCLAMATION = 48
   integer(C_INT), parameter :: MB_ICONWARNING = 48
   integer(C_INT), parameter :: MB_ICONINFORMATION = 64
   integer(C_INT), parameter :: MB_ICONASTERISK = 64
   integer(C_INT), parameter :: MB_ICONQUESTION = 32
   integer(C_INT), parameter :: MB_ICONSTOP = 16
   integer(C_INT), parameter :: MB_ICONERROR = 16
   integer(C_INT), parameter :: MB_ICONHAND = 16
   integer(C_INT), parameter :: MB_DEFBUTTON1 = 0
   integer(C_INT), parameter :: MB_DEFBUTTON2 = 256
   integer(C_INT), parameter :: MB_DEFBUTTON3 = 512
   integer(C_INT), parameter :: MB_DEFBUTTON4 = 768
   integer(C_INT), parameter :: MB_APPLMODAL = 0
   integer(C_INT), parameter :: MB_SYSTEMMODAL = 4096
   integer(C_INT), parameter :: MB_TASKMODAL = 8192
   integer(C_INT), parameter :: MB_DEFAULT_DESKTOP_ONLY = 131072
   integer(C_INT), parameter :: MB_RIGHT = 524288
   integer(C_INT), parameter :: MB_RTLREADING = 1048576
   integer(C_INT), parameter :: MB_SETFOREGROUND = 65536
   integer(C_INT), parameter :: MB_TOPMOST = 262144
   integer(C_INT), parameter :: MB_SERVICE_NOTIFICATION = 2097152
   interface
      function MultiByteToWideChar(CodePage,dwFlags,lpMultiByteStr, &
         cbMultiByte,lpWideCharStr,cchWideChar) bind(C,name='MultiByteToWideChar')
         import
         implicit none
         !GCC$ ATTRIBUTES STDCALL :: MultiByteToWideChar
         !DEC$ ATTRIBUTES STDCALL :: MultiByteToWideChar
         integer(C_INT) MultiByteToWideChar
         integer(UINT), value :: CodePage
         integer(DWORD), value :: dwFlags
         character(KIND=C_CHAR) lpMultiByteStr(*)
         integer(C_INT), value :: cbMultiByte
         integer(WCHAR) lpWideCharStr(*)
         integer(C_INT), value :: cchWideChar
      end function MultiByteToWideChar
   end interface
   interface
      function WideCharToMultiByte(CodePage,dwFlags,lpWideCharStr, &
         cchWideChar,lpMultiByteStr,cbMultiByte,lpDefaultChar, &
         lpUsedDefaultChar) bind(C,name='WideCharToMultiByte')
         import
         implicit none
         !GCC$ ATTRIBUTES STDCALL :: WideCharToMultiByte
         !DEC$ ATTRIBUTES STDCALL :: WideCharToMultiByte
         integer(C_INT) WideCharToMultiByte
         integer(UINT), value :: CodePage
         integer(DWORD), value :: dwFlags
         integer(WCHAR) lpWideCharStr(*)
         integer(C_INT), value :: cchWideChar
         character(KIND=C_CHAR) lpMultiByteStr(*)
         integer(C_INT), value :: cbMultiByte
         character(KIND=C_CHAR) lpDefaultChar
         integer(BOOL) lpUsedDefaultChar
      end function WideCharToMultiByte
   end interface
   interface
      function GetLastError() bind(C,name='GetLastError')
         import
         implicit none
         !GCC$ ATTRIBUTES STDCALL :: GetLastError
         !DEC$ ATTRIBUTES STDCALL :: GetLastError
         integer(DWORD) GetLastError
      end function GetLastError
   end interface
   interface
      function MessageBoxW(hWnd,lpTExt,lpCaption,uType) bind(C,name='MessageBoxW')
         import
         implicit none
         !GCC$ ATTRIBUTES STDCALL :: MessageBoxW
         !DEC$ ATTRIBUTES STDCALL :: MessageBoxW
         integer(C_INT) MessageBoxW
         integer(HANDLE), value :: hWnd
         integer(WCHAR) lpText(*)
         integer(WCHAR) lpCaption(*)
         integer(UINT), value :: uType
      end function MessageBoxW
   end interface
end module ifwina

program mbcs
   use ifwina
   use ISO_C_BINDING
   implicit none
   integer(C_INT) iret1
   integer(DWORD) iret2
   character(LEN=256,KIND=C_CHAR) lpMultiByteStr
   integer(WCHAR) lpWideCharStr(LEN(lpMultiByteStr))
   integer(UINT) CodePage
   integer(DWORD) dwFlags
   integer(C_INT) cbMultiByte
   integer(C_INT) cchWideChar
   character(KIND=C_CHAR), pointer :: lpDefaultChar
   integer(BOOL), pointer :: lpUsedDefaultChar
   integer(HANDLE) hWnd
   integer(WCHAR), pointer :: lpCaption(:)
   integer(UINT) uType
   character(KIND=C_CHAR), pointer :: lpNULL(:)

   CodePage = CP_UTF8
   dwFlags = MB_ERR_INVALID_CHARS
   lpMultiByteStr = 'Hello world'//ACHAR(0)
   cbMultiByte = -1
   lpWideCharStr = lpWideCharStr
   cchWideChar = size(lpWideCharStr)
   iret1 = MultiByteToWideChar(CodePage,dwFlags, &
      lpMultiByteStr,cbMultiByte,lpWideCharStr,cchWideChar)
   if(iret1 == 0) then
      iret2 = GetLastError()
      write(*,'(a,i0)') 'MultiByteToWideChar failed with error code ',iret2
   else
      write(*,'(a,i0,a)') 'MultiByteToWideChar converted ',iret1,' characters'
      hWnd = 0
      call C_F_POINTER(C_NULL_PTR,lpCaption,[0])
      uType = iany([MB_OK,MB_ICONINFORMATION])
      iret1 = MessageBoxW(hWnd,lpWideCharStr,lpCaption,uType)
      if(iret1 == 0) then
         iret2 = GetLastError()
         write(*,'(a,i0)') 'MessageBoxW failed with error code ',iret2
      else
         write(*,'(a,i0)') 'MessageBoxW returned code ', iret1
      end if
   end if
   lpMultiByteStr = ''
   dwFlags = 0
   cbMultiByte = 0
   call C_F_POINTER(C_NULL_PTR,lpDefaultChar)
   call C_F_POINTER(C_NULL_PTR,lpUsedDefaultChar)
   cchWideChar = -1
   call C_F_POINTER(C_NULL_PTR,lpNULL,[0])
   iret1 = WideCharToMultiByte(CodePage,dwFlags,lpWideCharStr,cchWideChar, &
      lpNULL,cbMultiByte,lpDefaultChar,lpUsedDefaultChar)
   if(iret1 == 0) then
      iret2 = GetLastError()
      write(*,'(a,i0)') 'WideCharToMultiByte failed with error code ',iret2
   else
      write(*,'(a,i0,a)') 'WideCharToMultiByte requires a ',iret1,' character buffer'
   end if
   dwFlags = WC_ERR_INVALID_CHARS
   cbMultiByte = len(lpMultiByteStr)
   iret1 = WideCharToMultiByte(CodePage,dwFlags,lpWideCharStr,cchWideChar, &
      lpMultiByteStr,cbMultiByte,lpDefaultChar,lpUsedDefaultChar)
   if(iret1 == 0) then
      iret2 = GetLastError()
      write(*,'(a,i0)') 'WideCharToMultiByte failed with error code ',iret2
   else
      write(*,'(a,i0,a)') 'MultiByteToWideChar converted ',iret1,' characters'
      write(*,'(2a)') 'lpMultiByteStr = ',lpMultiByteStr(1:index(lpMultiByteStr,ACHAR(0))-1)
   end if
end program mbcs

[Steven_L_Intel1]

RO is exactly right in his diagnosis. An easier workaround is to replace the NULL by %VAL(0) for the last argument.

I'll get this fixed in KERNEL32 (and will check the related declarations to see if there are similar problems.)

[JVanB]

Thinking about the situation a little more, maybe it's a weakness in the properties of !DEC$ ATTRIBUTES ALLOW_NULL. In a situation like this, if you want to pass FALSE (from ifwinty) as actual argument and ALLOW_NULL were in force, you would have to use %REF(FALSE) because the extension is documented to pass C_NULL_PTR by value if the actual argument is zero. As we have seen, if ALLOW_NULL is not in force and we want to pass C_NULL_PTR by value, we need %VAL(C_NULL_PTR). If we pass %VAL(0), does the ABI guarantee that the high 32 bits of the dummy argument on the stack are cleared to zeros in a 64-bit environment?

In any case it seems to be sort of a darned if you do, darned if you don't situation if the dummy argument is int* and (void *)0 and a pointer to zero are both acceptable arguments. Kind of like the problem with OPTIONAL, VALUE that led to ifort switching how it handled passing VALUE arguments to be 'by reference to a copy' instead of true value. I don't know how to fix this otherwise useful ALLOW_NULL extension and not be left with confusing cases where the compiler does something unexpected without warning unless you look at ifort's corresponding interface block as well as MSDN's documentation of the function each time you use the extension.

[Steven_L_Intel1]

The problem with VALUE was simply a misunderstanding of what the standard said. OPTIONAL had nothing to do with it. But your general point is well taken. Perhaps what we'll have to do is add a comment in kernel32.f90 explaining how to omit that argument (%VAL(0)).

[Steven_L_Intel1]

I was also a bit too hasty - ALLOW_NULL is exactly the right fix for this. Why isn't the situation RO describes a problem? Because this is an output argument - you'll never be passing a FALSE value IN to the routine here. You either supply the address of a BOOL-sized integer variable, or NULL. If you supply a constant NULL (or FALSE), it is't writable so you'll get the error anyway (which is what is happening here.)

With ALLOW_NULL, passing NULL means omitting the argument. Otherwise you have to pass a variable. I have escalated this to development as issue DPD200362551.

[JVanB]

RegQueryValueEx
XInputGetAudioDeviceIds
SetFilePointer
recvfrom
RegEnumKeyEx
CryptDecryptMessage and related...
WlanGetProfile
InternetSetFilePointer

[Steven_L_Intel1]

RO, have you looked at the documentation of those procedures to see if a problem actually exists? For example, in RegQueryValueEx I assume you're referring to lpcbData, which is an INOUT size of the lpData buffer. ALLOW_NULL, which is specified here, isn't a problem for the same reason as I describe above - you can't pass a constant 0 for this argument and expect anything useful.

[JVanB]

I only have a very old version of advapi32.f90 so I can't check from here. I just did a google search for Win32 API functions with intent inout and optional integer arguments and that's what it turned up. I can't check further without more significant effort here.

[Steven_L_Intel1]

Ok, I thought maybe you did that. Certainly those are potential problem candidates, but I haven't yet found other routines where an actual problem exists.