Latest IFX debugger problem (TOKENIZE example)

[andrew_4619]

Having loaded the latest release  from 31/10/2024 (2025.0.) I though I would try the TOKENIZE intrinsic having seen another  user post a problem on this:

As you can see we correctly write a bunch of herbs to the console but the debugger watch and hover show a whole load of parsley. The Scarborough Fair ballad with Parsley, Parley, Parsley and Parsley doesn't  have the same ring to it.

Thought I would mention this as IFX/Debugger issues are my main outstanding IFX issues. I know some have been fixed for this release.....

program test
    implicit none(type, external)
    external :: tokentest!, dataproc
    call tokentest()
    pause
    !call dataproc()
    !pause
end program test
subroutine tokentest()
    implicit none(type, external)
    CHARACTER(LEN=:),ALLOCATABLE,DIMENSION(:) :: tokens, separators
    CHARACTER(LEN=3):: delims = ',&'
    CHARACTER(LEN=:),ALLOCATABLE :: herbs
    INTEGER,ALLOCATABLE,DIMENSION(:)  :: begins, ends
    herbs  = 'parsley,sage,rosemary,&thyme'
    CALL TOKENIZE (herbs, delims, tokens, separators)
    write(*,'(A)') tokens
    write(*,'(A)') separators
    CALL TOKENIZE (herbs, delims, begins, ends)
end subroutine tokentest

[andrew_4619]

bump

[andrew_4619]

Still wrong in IFX 2025.1.0

[JohnNichols]

I think there may be an incorrect definition for tokens, as your model picks up single tokens, but not doubles. So ch in position 1 stuffs the code, but as last input, we just get C.  

[JohnNichols]

The code was copied directly from IFORT manual and it has perhaps an error.  Sorry maybe Steve knows, but at the moment the code thinks tokens has 1 character only.  

It will not compile in IFORT. 

[jimdempseyatthecove]

@JohnNichols 

Your code is not similar to @andrew_4619 

You should have herbs = '1,2,3,,CH'

Note the ",,"

Also

CHARACTER(LEN=:),ALLOCATABLE,DIMENSION(:) :: tokens, separators

May be generating a LEN=len_trim(first token)

to test, use herbs = 'CH,1,2,3'

EDIT - However, the above is a false assumption, as Andrew's output shows LEN of "rosemary" which is 8 as opposed to that of "parsley" which is 7.

Jim Dempsey

[andrew_4619]

I should add that the code I showed runs correctly it is only the debugger output that is wrong.

[jimdempseyatthecove]

It's odd that the debugger is wrong.

[JohnNichols]

The thing I noticed was the less than smiley face next to the debugger.  We are not interested in the console output.  

You can get the debugger working as long as the tokens are single atoms of data.  So it it clearly a known bug, but at least we know it works in part.  I thought it might have been in the Fortran code, but Andrew showed I was wrong. 

This is intel's problem, we can but wait for the debugger to get better.  

IFX debugger is buggy, but one is always able to use the console or the debug screen - which is really useful for stuff like this.  

Why the heck did they just not call it Split and not tokenize, Fortran is ?????????????? sometimes. 

[JohnNichols]

Andrew's code will not compile in IFORT. 

[andrew_4619]

The tokenize function was added in Fortran 2023. IFORT never supported it.  There has been a much used Fortran utility called split since the dawn of time. That takes a Fortran file and explodes each subroutine or function into its own individual source file. 

I raised this topic in the hope that this would get evaluated and maybe added to the list of debugger issues. Maybe it is just another form of a known issue that is in the pipeline. Who knows?

[JFH]

In F2023 split and tokenize are two different intrinsic subroutines. You can get the effect of tokenize by calling split as described in the F2023 standard 16.1.196 or "Modern Fortran Explained" 2023 edition 23.3.1.

[JohnNichols]

It is simple in C#

string line = "a lot of rubbish";

string[] words = line.Split(etc..) 

It would be nice if Fortran had some of these features.