Output of UTF-8 characters

[Paul_Felz]

Hi,

I have a problem with writing of UTF-8 encoded characters to disk. I need this for my language unfortunately has some non-ascii characters and As I user German data sourdes text items will have these 'Umlauts'. Here is my simple code:

!
!   Test Unicode UTF-8 input from file, handling and output to file
!
program UnicodeTest
implicit none

character*16 cString1, cString2
integer iRslt

open (unit = 10, fIle = 'Unicode_In.csv', encoding = 'UTF-8', iostat = iRslt)
read (10,'(a16)') cString1
cString2 = 'aäoöuüAÄOÖUÜszß'
open(unit = 11, file = 'Unicode_Out.csv', encoding = 'UTF-8')
write (11, '(a16)') cString1
write (11, '(a16)') cString2

stop
end

Unicode_In.csv contains the same string as cString2 defined in line #12:

aäoöuüAÄOÖUÜszß

the output-file is then

aäoöuüAÄOÖU
a䯶uüAď֕ܳzߠ

So while my input from file is processed properly (though censored), strings that are defined within the program seem not get encoded for output properly. And because my output will contain some explanatory text, I would like to have  it properly encoded as well.

Any Idea anybody?

[jimdempseyatthecove]

Not sure if this has anything to do with the issue.

UTF8 files generally have the UTF8 BOM (Byte Order Marking) as the first 3 bytes of the file.

See it is present (or required) to be in the file created.

Jim Dempsey

[andrew_4619]

You might read this..... https://fortran-lang.discourse.group/t/using-unicode-characters-in-fortran/2764

[jimdempseyatthecove]

@Paul_Felz 

Considering the link @andrew_4619 posted, your best bet would be to open the file as Stream.

Add the UTF8 BOM marker on first write

Check/skip the UTF8 BOM marker on first read

You will have to experiment with keeping/removing the encoding='UTF-8'

AND check to see if you have any nA format descriptors to confirm if/if not the n represents multi-byte characters as 1 byte or 2,3,4 bytes.

Jim Dempsey

[Paul_Felz]

Thanks to all of you.

The link Andrew gave was quite informative. But unfortunately this did not solve my problem, because my issue was not covered: Output of a non-ascii string generated within the program.

I tried with BOM markers but this did not solve the problem either.

Here is the code I tried in the end:

program UnicodeTest
implicit none

character*100 cString1, cString2
integer iRslt
character*3 cBOM

cBOM(1:1) = char(239)
cBOM(2:2) = char(187)
cBOM(3:3) = char(191)

open (unit = 10, fIle = 'Unicode_In.csv', encoding = 'UTF-8', iostat = iRslt)
read (10,'(a20)') cString1
cString2 = 'aäoöuüAÄOÖUÜszß'

open(unit = 11, file = 'Unicode_Out.csv', encoding = 'UTF-8')

write (11, *) cBOM, trim(cString1)
write (11, *) cBOM, trim(cString2)

stop
end

And regardless if I put this marker at the beginning of the output-lines or not, the strings remained the same as before.

I checked the input file - which was encoded as UTF-8 by using Libre-Office Calc - with a Hex-Editor and there was no such marker present. This is what Unicode_out.csv looks like

What puzzles me greatly is the fact that in the HEX-editor the first string is not legible but the second is - while in the output it is the nother way round.

What did I miss?

[MWind2]

Edited delete as not correct.

[MWind2]

cstring2 is written out as bytes, cstring1 input is UTF-8.

Some code used to make UTF-16LE included; Windows c dll and Fortran exe.

[Devorah_H_Intel]

---

@MWind2 wrote:

cstring2 is written out as bytes, cstring1 input is UTF-8.

Some code used to make UTF-16LE included; Windows c dll and Fortran exe.

---

Is it possible to include all in one zip file? 

[MWind2]

Sorry, OK.

[GVautier]

Hello

First : The BOM must be written only once at the beginning of the file. It indicates that the content of the file must be interpreted as UTF8. The BOM is not mandatory, some editors like Notepad++ allow automatic encoding detection and selection of the desired character encoding.

Second : By using * format specifier, I think that a space is inserted at the beginning of each line and so the BOM cannot be interpreted.

Third : I think that your character constant is encoded as your source file. So, if true, your source file must be encoded in UTF8.

Fourth : An UTF8 string  is longer than the number of displayed characters. So you must oversize your character variables from 16 to 32 minimum to avoid truncation.

[Paul_Felz]

Thanks MWind2

Because it is a very limited number of characters only that give me the headaches, I added a few lines to my input-routine, that replaces two-byte entries for the special characters to the single internal one. On output I will do just vice versa (not yet completed). Here is my code, that I use to parse my semicolon-seperated input lines. :

character*1024 Function cParser(cLine)
    implicit none
    character*1024 cLine
    character*2, dimension(7) :: cOld
    character*1, dimension(7) :: cNew
    integer j
    integer iPos
!** Characters to be replaced
    cOld(1) = char(195)//char(164)      ! ä
    cOld(2) = char(195)//char(182)      ! ö
    cOld(3) = char(195)//char(188)      ! ü
    cOld(4) = char(195)//char(132)      ! Ä
    cOld(5) = char(195)//char(150)      ! Ö
    cOld(6) = char(195)//char(156)      ! Ü
    cOld(7) = char(195)//char(159)      ! ß
!*** by these literals
    cNew(1) = char(228)
    cNew(2) = char(246)
    cNew(3) = char(252)
    cNew(4) = char(196)
    cNew(5) = char(214)
    cNew(6) = char(220)
    cNew(7) = char(223)
 
!*** find next ;
    iPos = index (cLine,';')
!*** if there is none - return rest of line and scratch it.
    if (iPos .eq. 0) then
        cParser = trim(cLine)
        cLine = ''
        return
!*** if it is the first sign, return empty string and cut off ;
    else if (iPos .eq. 1) then
        cParser = ''
        cLine = cLine (iPos + 1 :)
        return
!*** If it is anywhere in the record, process the string up to the ;
    else
        cParser = adjustl (cLine (1 : iPos - 1))
        !*** discard the read part from the input record
        cLine = cLine (iPos + 1 :)
        !*** Replace special characters until there are no more
        do j = 1, 7
            do
                iPos = index (cParser, cOld(j))
                if (iPos .eq. 0) exit
                cParser = cParser (: iPos - 1) // cNew(j) // cParser (iPos + 2 :)
            enddo
        enddo               
    endif
    return
end