- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I have a problem with writing of UTF-8 encoded characters to disk. I need this for my language unfortunately has some non-ascii characters and As I user German data sourdes text items will have these 'Umlauts'. Here is my simple code:
!
! Test Unicode UTF-8 input from file, handling and output to file
!
program UnicodeTest
implicit none
character*16 cString1, cString2
integer iRslt
open (unit = 10, fIle = 'Unicode_In.csv', encoding = 'UTF-8', iostat = iRslt)
read (10,'(a16)') cString1
cString2 = 'aäoöuüAÄOÖUÜszß'
open(unit = 11, file = 'Unicode_Out.csv', encoding = 'UTF-8')
write (11, '(a16)') cString1
write (11, '(a16)') cString2
stop
end
Unicode_In.csv contains the same string as cString2 defined in line #12:
aäoöuüAÄOÖUÜszß
the output-file is then
aäoöuüAÄOÖU
a䯶uüAď֕ܳzߠ
So while my input from file is processed properly (though censored), strings that are defined within the program seem not get encoded for output properly. And because my output will contain some explanatory text, I would like to have it properly encoded as well.
Any Idea anybody?
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Not sure if this has anything to do with the issue.
UTF8 files generally have the UTF8 BOM (Byte Order Marking) as the first 3 bytes of the file.
See it is present (or required) to be in the file created.
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You might read this..... https://fortran-lang.discourse.group/t/using-unicode-characters-in-fortran/2764
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Considering the link @andrew_4619 posted, your best bet would be to open the file as Stream.
Add the UTF8 BOM marker on first write
Check/skip the UTF8 BOM marker on first read
You will have to experiment with keeping/removing the encoding='UTF-8'
AND check to see if you have any nA format descriptors to confirm if/if not the n represents multi-byte characters as 1 byte or 2,3,4 bytes.
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks to all of you.
The link Andrew gave was quite informative. But unfortunately this did not solve my problem, because my issue was not covered: Output of a non-ascii string generated within the program.
I tried with BOM markers but this did not solve the problem either.
Here is the code I tried in the end:
program UnicodeTest
implicit none
character*100 cString1, cString2
integer iRslt
character*3 cBOM
cBOM(1:1) = char(239)
cBOM(2:2) = char(187)
cBOM(3:3) = char(191)
open (unit = 10, fIle = 'Unicode_In.csv', encoding = 'UTF-8', iostat = iRslt)
read (10,'(a20)') cString1
cString2 = 'aäoöuüAÄOÖUÜszß'
open(unit = 11, file = 'Unicode_Out.csv', encoding = 'UTF-8')
write (11, *) cBOM, trim(cString1)
write (11, *) cBOM, trim(cString2)
stop
end
And regardless if I put this marker at the beginning of the output-lines or not, the strings remained the same as before.
I checked the input file - which was encoded as UTF-8 by using Libre-Office Calc - with a Hex-Editor and there was no such marker present. This is what Unicode_out.csv looks like
What puzzles me greatly is the fact that in the HEX-editor the first string is not legible but the second is - while in the output it is the nother way round.
What did I miss?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
cstring2 is written out as bytes, cstring1 input is UTF-8.
Some code used to make UTF-16LE included; Windows c dll and Fortran exe.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@MWind2 wrote:
cstring2 is written out as bytes, cstring1 input is UTF-8.
Some code used to make UTF-16LE included; Windows c dll and Fortran exe.
Is it possible to include all in one zip file?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello
First : The BOM must be written only once at the beginning of the file. It indicates that the content of the file must be interpreted as UTF8. The BOM is not mandatory, some editors like Notepad++ allow automatic encoding detection and selection of the desired character encoding.
Second : By using * format specifier, I think that a space is inserted at the beginning of each line and so the BOM cannot be interpreted.
Third : I think that your character constant is encoded as your source file. So, if true, your source file must be encoded in UTF8.
Fourth : An UTF8 string is longer than the number of displayed characters. So you must oversize your character variables from 16 to 32 minimum to avoid truncation.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks MWind2
Because it is a very limited number of characters only that give me the headaches, I added a few lines to my input-routine, that replaces two-byte entries for the special characters to the single internal one. On output I will do just vice versa (not yet completed). Here is my code, that I use to parse my semicolon-seperated input lines. :
character*1024 Function cParser(cLine)
implicit none
character*1024 cLine
character*2, dimension(7) :: cOld
character*1, dimension(7) :: cNew
integer j
integer iPos
!** Characters to be replaced
cOld(1) = char(195)//char(164) ! ä
cOld(2) = char(195)//char(182) ! ö
cOld(3) = char(195)//char(188) ! ü
cOld(4) = char(195)//char(132) ! Ä
cOld(5) = char(195)//char(150) ! Ö
cOld(6) = char(195)//char(156) ! Ü
cOld(7) = char(195)//char(159) ! ß
!*** by these literals
cNew(1) = char(228)
cNew(2) = char(246)
cNew(3) = char(252)
cNew(4) = char(196)
cNew(5) = char(214)
cNew(6) = char(220)
cNew(7) = char(223)
!*** find next ;
iPos = index (cLine,';')
!*** if there is none - return rest of line and scratch it.
if (iPos .eq. 0) then
cParser = trim(cLine)
cLine = ''
return
!*** if it is the first sign, return empty string and cut off ;
else if (iPos .eq. 1) then
cParser = ''
cLine = cLine (iPos + 1 :)
return
!*** If it is anywhere in the record, process the string up to the ;
else
cParser = adjustl (cLine (1 : iPos - 1))
!*** discard the read part from the input record
cLine = cLine (iPos + 1 :)
!*** Replace special characters until there are no more
do j = 1, 7
do
iPos = index (cParser, cOld(j))
if (iPos .eq. 0) exit
cParser = cParser (: iPos - 1) // cNew(j) // cParser (iPos + 2 :)
enddo
enddo
endif
return
end
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page