topic Re: Latest IFX debugger problem (TOKENIZE example) in Intel® Fortran Compiler

Latest IFX debugger problem (TOKENIZE example)

andrew_4619 — Fri, 01 Nov 2024 20:42:32 GMT

Having loaded the latest release from 31/10/2024 (2025.0.) I though I would try the TOKENIZE intrinsic having seen another user post a problem on this:

As you can see we correctly write a bunch of herbs to the console but the debugger watch and hover show a whole load of parsley. The Scarborough Fair ballad with Parsley, Parley, Parsley and Parsley doesn't have the same ring to it.

Thought I would mention this as IFX/Debugger issues are my main outstanding IFX issues. I know some have been fixed for this release.....

program test implicit none(type, external) external :: tokentest!, dataproc call tokentest() pause !call dataproc() !pause end program test subroutine tokentest() implicit none(type, external) CHARACTER(LEN=:),ALLOCATABLE,DIMENSION(:) :: tokens, separators CHARACTER(LEN=3):: delims = ',&' CHARACTER(LEN=:),ALLOCATABLE :: herbs INTEGER,ALLOCATABLE,DIMENSION(:) :: begins, ends herbs = 'parsley,sage,rosemary,&thyme' CALL TOKENIZE (herbs, delims, tokens, separators) write(*,'(A)') tokens write(*,'(A)') separators CALL TOKENIZE (herbs, delims, begins, ends) end subroutine tokentest

Re: Latest IFX debugger problem (TOKENIZE example)

andrew_4619 — Mon, 25 Nov 2024 15:17:50 GMT

bump

Re: Latest IFX debugger problem (TOKENIZE example)

andrew_4619 — Sun, 04 May 2025 15:58:02 GMT

Still wrong in IFX 2025.1.0

Re: Latest IFX debugger problem (TOKENIZE example)

JohnNichols — Sun, 04 May 2025 16:17:16 GMT

I think there may be an incorrect definition for tokens, as your model picks up single tokens, but not doubles. So ch in position 1 stuffs the code, but as last input, we just get C.

Re: Latest IFX debugger problem (TOKENIZE example)

JohnNichols — Sun, 04 May 2025 16:21:47 GMT

The code was copied directly from IFORT manual and it has perhaps an error. Sorry maybe Steve knows, but at the moment the code thinks tokens has 1 character only.

It will not compile in IFORT.

Re: Latest IFX debugger problem (TOKENIZE example)

andrew_4619 — Sun, 04 May 2025 17:29:35 GMT

I should add that the code I showed runs correctly it is only the debugger output that is wrong.

Re: Latest IFX debugger problem (TOKENIZE example)

jimdempseyatthecove — Mon, 05 May 2025 13:11:23 GMT

@JohnNichols

Your code is not similar to @andrew_4619

You should have herbs = '1,2,3,,CH'

Note the ",,"

Also

CHARACTER(LEN=:),ALLOCATABLE,DIMENSION(:) :: tokens, separators

May be generating a LEN=len_trim(first token)

to test, use herbs = 'CH,1,2,3'

EDIT - However, the above is a false assumption, as Andrew's output shows LEN of "rosemary" which is 8 as opposed to that of "parsley" which is 7.

Jim Dempsey

Re: Latest IFX debugger problem (TOKENIZE example)

jimdempseyatthecove — Mon, 05 May 2025 13:08:24 GMT

It's odd that the debugger is wrong.

Re: Latest IFX debugger problem (TOKENIZE example)

JohnNichols — Mon, 05 May 2025 17:12:48 GMT

The thing I noticed was the less than smiley face next to the debugger. We are not interested in the console output.

You can get the debugger working as long as the tokens are single atoms of data. So it it clearly a known bug, but at least we know it works in part. I thought it might have been in the Fortran code, but Andrew showed I was wrong.

This is intel's problem, we can but wait for the debugger to get better.

IFX debugger is buggy, but one is always able to use the console or the debug screen - which is really useful for stuff like this.

Why the heck did they just not call it Split and not tokenize, Fortran is ?????????????? sometimes.

Re: Latest IFX debugger problem (TOKENIZE example)

JohnNichols — Mon, 05 May 2025 17:13:31 GMT

Andrew's code will not compile in IFORT.

Re: Latest IFX debugger problem (TOKENIZE example)

andrew_4619 — Mon, 05 May 2025 21:38:14 GMT

The tokenize function was added in Fortran 2023. IFORT never supported it. There has been a much used Fortran utility called split since the dawn of time. That takes a Fortran file and explodes each subroutine or function into its own individual source file.

I raised this topic in the hope that this would get evaluated and maybe added to the list of debugger issues. Maybe it is just another form of a known issue that is in the pipeline. Who knows?

Re: Latest IFX debugger problem (TOKENIZE example)

JFH — Mon, 05 May 2025 23:03:28 GMT

In F2023 split and tokenize are two different intrinsic subroutines. You can get the effect of tokenize by calling split as described in the F2023 standard 16.1.196 or "Modern Fortran Explained" 2023 edition 23.3.1.

Re: Latest IFX debugger problem (TOKENIZE example)

JohnNichols — Mon, 05 May 2025 23:12:34 GMT

It is simple in C#

string line = "a lot of rubbish";

string[] words = line.Split(etc..)

It would be nice if Fortran had some of these features.