Dependency analysis of mild insanity in Fortran

[IanH]

For those who want to have a bit of a giggle...

A while back, as a bit of a F2003 learning exercise I started writing a little Fortran parser in Fortran that lets you do dependency analysis - "tell me what order I should compile my files".  If you are working in Visual Studio this sort of thing is done automagically for you, but if you ever have to drop back to the command line then this sort of utility actually can come in handy.  And sometimes it even gives the right output.

Because it was a learning exercise crossed with delusions of grandeur and a good dash of a tasty cabernet/merlot blend it has been put together from "first principles" - some script kiddy could probably achieve the same in a dozen lines of the scripting language of the day using a few regexes and a hash thingamajig, but honestly people - where is the fun in that?  Plus it does F2008 syntax And Nothing More (it was supposed to be a F2003 learning exercise, but I learn real slow).  And you need fairly recent ifort, because writing this has, at times over the years, felt rather like a ifort compiler testing exercise.

So, if you compile the attached (you need /standard-semantics) , you can do things like the following from the cmd prompt (with extensions enabled):

[plain]>>set files= & (for /f "usebackq" %f in (`dir *.f90 /b ^| FF08Depends -`) do @(echo %f & ifort /c /nologo /standard-semantics %f & set files=!files! %~nf.obj)) & ifort /nologo /FeFF08Depends.exe !files!
CharUtils.f90
Strings.f90
ErrorLocations.f90
ErrorLevels.f90
Errors.f90
CompilerKinds.f90
Sources.f90
SourceLocations.f90
Tokens.f90
MatchUtils.f90
UseStmtsUtils.f90
UnitUtilities.f90
SyntaxParts.f90
SubmoduleStmtsUtils.f90
StmtFunctionStmtsUtils.f90
Statements.f90
StatementData.f90
SourceFiles.f90
ErrorCodes.f90
CharacterTypes.f90
Scanner.f90
ScanFreeForm.f90
ScanFixedForm.f90
ProgramStmtsUtils.f90
ModuleStmtsUtils.f90
IntegerLists.f90
IncludeLines.f90
FileUtilities.f90
Debug.f90
CompilerHosts.f90
Classifier.f90
FileDependencyPass.f90
DependencyUtils.f90
DefaultHosts.f90
DependsImplementation.f90
CmdLine.f90
FF08Depends.f90[/plain]

(which shows you how to compile and link the tool that you need to compile and link the tool that shows you how to compile and link the tool - and to think Fortran previously couldn't handle recursion...) 

Then combine it with the graphviz/dot tool, and you can also generate lots of wacky diagrams:

Now, for some inexplicable reason I think I might go and have spaghetti for dinner.

[IanH]

(Follow-up because the forum... etc.)
 

[Les_Neilson]

I shall tell my boss that the reason development is so slow is a lack of good quality (or indeed any) cabernet / merlot :-)

We do occasionally run to donuts but my doctor has advised me against taking in so much sugar :-(

Les

[FortranFan]

Very interesting, Ian.

"If you are working in Visual Studio this sort of thing is done automagically for you, but if you ever have to drop back to the command line then this sort of utility actually can come in handy"

So now that you've done this, what is your impression of Visual Studio's performance on sorting out code dependencies?  If you analyze some test code by yourself and with your utility and then build in Visual Studio, do you feel Visual Studio does just as well as parsing it as your utility?  Or is it poorer?

Thanks,

[Paul_Curtis]

I have very complex multi-module (80+) projects, and find the IVF dependency checker in VS now works very well, this was not always the case.  Cautionary note: the dependency checker often fails when old projects are imported into VS, but works when a new project is created and the source files are explicitly added into the project.

Having struggled for many years through MS/DVF/Compaq fortrans which lacked an IDE and required extensive use of multilevel overlays to make the code fit and to avoid dependency problems and circularities, I would never go backwards.  VS is wonderful (IMHO).

[IanH]

I'm not aware of any current material deficiencies in the VS integration dependency analysis.  That said, I haven't tested it out with deliberate obfuscation attempts, such as splitting the use keyword token over multiple lines or similar nonsense.  I suspect performance wise the VS integration approach is vastly superior - it s pretty easy to pick out USE statements and MODULE statements (and then parse out module names) to a reasonable degree of reliability with applications of regular expressions or similar.  And like Paul - I live in VS most of the day (and perhaps too much of the night) - no way I'm going back to a basic text editor + command line for real work.

The specific useful applications include:

- The VS integration won't tell you outright when you have circular dependencies.  These days it is easy enough to recognize the symptoms ("why is my project always rebuilding?") but then you have to find the program units involved in the loop.

- You can't ask the VS integration what the dependencies are for a specific file.  I find this comes up from time to time as part of chopping down a larger program to a minimal compilable example in order to report a compiler bug or similar.

- The VS integration won't help you draw horrendously confusing charts that make you laugh.

And the VS integration probably doesn't do analysis for submodules :(

[FortranFan]

I agree overall on the value of Visual Studio.  I too use it extensively; in fact, I've moved to Visual Studio IDE since version 6.0 (was it circa 1998-99?) and I don't miss the days of makefiles, etc. that I used to use on Unix.  Apart from all the mixed-mode debugging performance issues in VS2012 and deviating completely away from any sense of aesthetics and ergonomics I have with versions VS2012 and VS2013, Microsoft rocks with this product! :-)

The reason for my question was somewhat along all what Ian mentioned, "why is my project always rebuilding?".  In my case, it is more like why some indirectly related, but not directly dependent, files always recompiling.  My projects are not that big and only take a few minutes to compile, so I'm not bothered too much.  But I was "extrapolating" and thinking, gee if I was working on some large projects, this would drive me nuts.

An example scenario of a broken dependency is as follows: an abstract derived type and a concrete type of this abstract in file (module) A, an extended type B of the concrete type A in file (module) B, an extended type C of the concrete type A in file (module) C - I now make a minor modification, even a comment line change, in file C and all 3 files recompile.  Admittedly, this is on an older Intel Fortran version so the issue may be fixed in more recent offerings.  If not, then I would have thought Ian et al. would be running into this a lot and that is what might have motivated him to create the above test utility.

I'll see if I can put together a simple test on my problem that I can share on this forum.

And the VS integration probably doesn't do analysis for submodules :(

- submodules can't come soon enough!  Hope Intel developers are already on it even if no date will be given for this feature any time soon!

Cheers,

[IanH]

FortranFan wrote:

An example scenario of a broken dependency is as follows: an abstract derived type and a concrete type of this abstract in file (module) A, an extended type B of the concrete type A in file (module) B, an extended type C of the concrete type A in file (module) C - I now make a minor modification, even a comment line change, in file C and all 3 files recompile.

I suspect that you have circular dependencies, perhaps involving module D, E or F that might not be immediately apparent.  Does everything build in one go after a clean?

If you compile the utility in the zip file in #2 (create a console program project for it, add the files from the zip, set the property Fortran > Language > Enable F2003 standard semantics to Yes and build it) and then supply it a list of all the source files in your program it [perhaps] will tell you if you have a dependency issue.  If all the files for your fortran program are in the one folder you can do something like:

[plain]dir *.f90 /b | FF08Depends -[/plain]

If things are ok it should just list the files supplied to it in a reasonable compilation order.  If there are circular dependencies between files or within a file you should get a diagnostic.  But if you are doing anything outside the remit of the standard (like using the C preprocessor to #define or #include stuff) then the utility will probably just get confused.  Note the qualifiers "should", "perhaps", etc are all in there for good reasons - this might also just waste your time.