topic Output of UTF-8 characters in Intel® Fortran Compiler

Output of UTF-8 characters

Paul_Felz — Sat, 22 Jun 2024 17:19:12 GMT

Hi,

I have a problem with writing of UTF-8 encoded characters to disk. I need this for my language unfortunately has some non-ascii characters and As I user German data sourdes text items will have these 'Umlauts'. Here is my simple code:

! ! Test Unicode UTF-8 input from file, handling and output to file ! program UnicodeTest implicit none character*16 cString1, cString2 integer iRslt open (unit = 10, fIle = 'Unicode_In.csv', encoding = 'UTF-8', iostat = iRslt) read (10,'(a16)') cString1 cString2 = 'aäoöuüAÄOÖUÜszß' open(unit = 11, file = 'Unicode_Out.csv', encoding = 'UTF-8') write (11, '(a16)') cString1 write (11, '(a16)') cString2 stop end

Unicode_In.csv contains the same string as cString2 defined in line #12:

aäoöuüAÄOÖUÜszß

the output-file is then

aäoöuüAÄOÖU a䯶uüAď֕ܳzߠ

So while my input from file is processed properly (though censored), strings that are defined within the program seem not get encoded for output properly. And because my output will contain some explanatory text, I would like to have it properly encoded as well.

Any Idea anybody?

Re: Output of UTF-8 characters

jimdempseyatthecove — Sat, 22 Jun 2024 18:32:51 GMT

Not sure if this has anything to do with the issue.

UTF8 files generally have the UTF8 BOM (Byte Order Marking) as the first 3 bytes of the file.

See it is present (or required) to be in the file created.

Jim Dempsey

Re: Output of UTF-8 characters

andrew_4619 — Sat, 22 Jun 2024 22:51:57 GMT

You might read this..... https://fortran-lang.discourse.group/t/using-unicode-characters-in-fortran/2764

Re: Output of UTF-8 characters

jimdempseyatthecove — Sun, 23 Jun 2024 12:48:45 GMT

@Paul_Felz

Considering the link @andrew_4619 posted, your best bet would be to open the file as Stream.

Add the UTF8 BOM marker on first write

Check/skip the UTF8 BOM marker on first read

You will have to experiment with keeping/removing the encoding='UTF-8'

AND check to see if you have any nA format descriptors to confirm if/if not the n represents multi-byte characters as 1 byte or 2,3,4 bytes.

Jim Dempsey

Re: Output of UTF-8 characters

Paul_Felz — Sun, 23 Jun 2024 17:35:18 GMT

Thanks to all of you.

The link Andrew gave was quite informative. But unfortunately this did not solve my problem, because my issue was not covered: Output of a non-ascii string generated within the program.

I tried with BOM markers but this did not solve the problem either.

Here is the code I tried in the end:

program UnicodeTest implicit none character*100 cString1, cString2 integer iRslt character*3 cBOM cBOM(1:1) = char(239) cBOM(2:2) = char(187) cBOM(3:3) = char(191) open (unit = 10, fIle = 'Unicode_In.csv', encoding = 'UTF-8', iostat = iRslt) read (10,'(a20)') cString1 cString2 = 'aäoöuüAÄOÖUÜszß' open(unit = 11, file = 'Unicode_Out.csv', encoding = 'UTF-8') write (11, *) cBOM, trim(cString1) write (11, *) cBOM, trim(cString2) stop end

And regardless if I put this marker at the beginning of the output-lines or not, the strings remained the same as before.

I checked the input file - which was encoded as UTF-8 by using Libre-Office Calc - with a Hex-Editor and there was no such marker present. This is what Unicode_out.csv looks like

What puzzles me greatly is the fact that in the HEX-editor the first string is not legible but the second is - while in the output it is the nother way round.

What did I miss?

Re: Output of UTF-8 characters

MWind2 — Mon, 24 Jun 2024 05:42:23 GMT

Edited delete as not correct.

Re: Output of UTF-8 characters

MWind2 — Tue, 25 Jun 2024 16:11:37 GMT

cstring2 is written out as bytes, cstring1 input is UTF-8.

Some code used to make UTF-16LE included; Windows c dll and Fortran exe.

Re: Output of UTF-8 characters

Devorah_H_Intel — Tue, 25 Jun 2024 20:35:15 GMT

@MWind2 wrote:

cstring2 is written out as bytes, cstring1 input is UTF-8.

Some code used to make UTF-16LE included; Windows c dll and Fortran exe.

Is it possible to include all in one zip file?

Re: Output of UTF-8 characters

Paul_Felz — Wed, 26 Jun 2024 15:57:45 GMT

Thanks MWind2

Because it is a very limited number of characters only that give me the headaches, I added a few lines to my input-routine, that replaces two-byte entries for the special characters to the single internal one. On output I will do just vice versa (not yet completed). Here is my code, that I use to parse my semicolon-seperated input lines. :

character*1024 Function cParser(cLine) implicit none character*1024 cLine character*2, dimension(7) :: cOld character*1, dimension(7) :: cNew integer j integer iPos !** Characters to be replaced cOld(1) = char(195)//char(164) ! ä cOld(2) = char(195)//char(182) ! ö cOld(3) = char(195)//char(188) ! ü cOld(4) = char(195)//char(132) ! Ä cOld(5) = char(195)//char(150) ! Ö cOld(6) = char(195)//char(156) ! Ü cOld(7) = char(195)//char(159) ! ß !*** by these literals cNew(1) = char(228) cNew(2) = char(246) cNew(3) = char(252) cNew(4) = char(196) cNew(5) = char(214) cNew(6) = char(220) cNew(7) = char(223) !*** find next ; iPos = index (cLine,';') !*** if there is none - return rest of line and scratch it. if (iPos .eq. 0) then cParser = trim(cLine) cLine = '' return !*** if it is the first sign, return empty string and cut off ; else if (iPos .eq. 1) then cParser = '' cLine = cLine (iPos + 1 :) return !*** If it is anywhere in the record, process the string up to the ; else cParser = adjustl (cLine (1 : iPos - 1)) !*** discard the read part from the input record cLine = cLine (iPos + 1 :) !*** Replace special characters until there are no more do j = 1, 7 do iPos = index (cParser, cOld(j)) if (iPos .eq. 0) exit cParser = cParser (: iPos - 1) // cNew(j) // cParser (iPos + 2 :) enddo enddo endif return end

Re: Output of UTF-8 characters

GVautier — Thu, 27 Jun 2024 12:27:19 GMT

Hello

First : The BOM must be written only once at the beginning of the file. It indicates that the content of the file must be interpreted as UTF8. The BOM is not mandatory, some editors like Notepad++ allow automatic encoding detection and selection of the desired character encoding.

Second : By using * format specifier, I think that a space is inserted at the beginning of each line and so the BOM cannot be interpreted.

Third : I think that your character constant is encoded as your source file. So, if true, your source file must be encoded in UTF8.

Fourth : An UTF8 string is longer than the number of displayed characters. So you must oversize your character variables from 16 to 32 minimum to avoid truncation.

Re: Output of UTF-8 characters

MWind2 — Thu, 27 Jun 2024 13:51:24 GMT

Sorry, OK.