Interesting Observations

[JohnNichols]

I tumbled across this 

Autocorrelation in Fortran

Well if we are going to be analyzing large sets of random data we cant do it in matlab because matlab is SUPER slow

so i wrote some autocorrelation in fortran. fortran is terrible at a lot of modern programming stuff, but its fast. For example, it is a major pain to set a parameter from a command line argument, or to get the length of a file. I am not a master at fortran, so i might just be incompetent but believe me it seems like too much of a pain.

So i have to set the autocorrelation dimension and data set size manually before compiling it. but anyways, here it is:

PROGRAM bob
!       USE OMP_lib
        INTEGER, PARAMETER :: P=140067,M=70000
        REAL(8) :: sum,out(M),data(P),VAR,P2
        INTEGER :: k,j
	P2 = P;
        open (unit = 2, file = "data.txt")	
        do k=1,P
                read(2,*) data(k)
        end do
	sum=0
	do k=1,P
                sum = sum + data(k)
        end do
	u = sum/P
        sum = 0
        do k=1,P
               sum = sum + (data(k)-u)*(data(k)-u)
        end do
	VAR = sum/(P-1)
	write (*,*) "STANDARD ERROR:"
	write (*,*) 1.96/SQRT(P2)
        do k=1,M
                out(k) = 0
                do j=k+1,P
                        out(k) = out(k) + (data(j)-u)*(data(j-k)-u);
                end do
		out(k) = out(k)/(VAR*P)
        end do
        open (unit = 7, file = "CORR.txt")
        do k=1,M
                write (7,*) out(k)
        end do
        close(7)
        END PROGRAM

the parameter P is the length of the file data.txt, i find it with wc -l. M is the dimension, must be less than half the length of the data set. it outputs a file CORR.txt which is the autocorrelation data to let you load it in matlab with data = load(‘CORR.txt’). As of now, the input file is a list of numbers seperated by newlines

it rips through the ~140000 element set in a few seconds

future directions will be making it so it can read a binary file and so i dont have to set the damn set length manually and set the correleation dimension at the command line. 

NO comment provided just his comments.

John

[JohnNichols]

https://math4u.wordpress.com/2009/09/30/autocorrelation-in-fortran/

Location of code

[FortranFan]

@John Nichols,

Your original post has considerable content that veers heavily toward striving to do expert commentary but which is so questionable that you need to first expend the time and energy to truly master Fortran before you include such content.

1. Take the time to go through each and every reference in the Dr Fortran blog on modern Fortran thoroughly: https://software.intel.com/en-us/blogs/2013/12/30/doctor-fortran-in-its-a-modern-fortran-world
2. Based on how many folks in the computer programming domains now view and try to benchmark program performance, Fortran is no faster at anything really.  See http://benchmarksgame.alioth.debian.org/u64q/which-programs-are-fastest.html for one example.
3. Modern Fortran has its gaps but you need to be specific rather than make a blanket statement such as " fortran is terrible at a lot of modern programming stuff".
4. See https://software.intel.com/en-us/fortran-compiler-18.0-developer-guide-and-reference-get-command-argument- it's as easy as anything else in Fortran to get command-line argument information; your statement, " it is a major pain to set a parameter from a command line argument"  is rather unfortunate.

[JohnNichols]

Dear Fortran Fan:

I stumbled across this work on a web site this morning. After I got thru chuckling I thought the Forum might like to see it - more by way of humour at the comments than anything else.

I would rewrite it -- but what is the point, the real shame is that everyone in my current work place thinks you do everything MATLAB or Maple or EXCEL and do not know they are being short changed. 

Bring back real engineers

John

[FortranFan]

Nichols, John wrote:

.. Bring back real engineers ..

@John Nichols,

Sorry, your comments are incomprehensible.

Humor is alright but not at the cost of an unsuspecting reader going away with the entirely false impression from tidbits such as "fortran is terrible at a lot of modern programming stuff", and then going on to even quote a PROFESSOR  from an institution of good repute to boot.  Come on, you need to do better, much better.

And what does anything here have to do with engineers anyway, seriously?  A 'real' engineer brings RIGOR to the task while keeping OVERALL EFFICIENCY in mind, not just settling for intermediate benefits (ala premature optimization as the evil decried by Knuth).  Your approach with Fortran shows none of this, starting with the missing 'IMPLICIT NONE' statement and a whole bunch of other deviations from good coding practices.  And in this day and age where preschool kids work with graphics on ever powerful electronic devices in 3D and like to explore and examine ever so closely patterns and behavior visually, your presenting of output in silly little 'CORR.txt' may involve lot more work to comsume the results fully than those using MATLAB and/or EXCEL.  Please realize again you are making no sense whatsoever.

[JohnNichols]

Dear Fortran Fan:

Thank you for your well thought out comments.

We may be at slightly cross purposes - so as clearly as an Australian can say anything:

1. I did not write the program or the comments. I thought these original comments were indicative of a lot of poorly researched data and programs one can find on the Internet. We have gone from the Publishers being reasonably selective about what they publish and usually but not always making a reasonable effort at fact checking - one can of course think of the old poorly fact checked books - to random garbage on a lot of websites - the Internet for Fools.  Interestingly Wikipedia seems to be at the publisher standard on most engineering matters.

2. I posted them on the Forum as they contain a small amount of humour in the comments, and from the fact that the writer failed to do even cursory checking of the Internet to see how easy it is to either solve the problems or find such things as this forum who have real experts  who can answer these questions in a few minutes.  Clearly a writer who is not doing a simple reference check is a poor writer.

3. In the olden days, say 1980, a Civil Engineering degree typically included a course or so based on Fortran. I had 2 - one in 1977 purely on Fortran and one in 1979 based on the standard Conte and de Boore book. Which is now available online.  It was hard but it set up one up for a life of programing and data analysis. Most engineers went on to avoid Fortran, but at least they were aware of it as they plugged away with limited data in EXCEL, but in 1980 we only had limited data.  

4. Now my students who are working on very large data sets, 3.4 million real data points every 8 minutes in one file, they want to use EXCEL or MATLAB as their professor(s) use EXCEL and  MATLAB, I would opine for the ease of teaching.  So that Felippa's novel 18 dof 3 node plate element originally written Fortran is now available and taught in MATLAB.  

% AUTHOR C. A. Felippa, June 1984 % Converted to Matlab from Mathematica, by S. Golmon, N. Ledford, % L. Liu, April 2009 % VERSION June 1984 - see attachment.

I asked Felippa for the Fortran code and he provided some of it, but what happens once he retires.  We need to rewrite from MATLAB to Fortran, which I did and this forum helped with the hard bits.

So now I am working with two very well educated engineering graduate students at the moment, they will both be full professors if they want or rich consultants. We are looking at correlation in signals, there are wonderful libraries in MKL Fortran that do so much and the code to implement what we want to do is say 300 to 400 lines - a solid days work.  They can use MATLAB really well and use a lot of routines, which is good for checking one signal of short length, now try continuous signals at 2000 data sets per second. But they have been taught MATLAB, not Fortran. So as far as I am concerned they are not prepared for a real world that is now solid wall to wall data. If you want to analyse it fast you can use C or Fortran. Ok, some like Python, but I just do not like the syntax.  I could do what I want to do in LISP in a lot less than 400 lines, but I do not have a week to wait for the answer.

So my students are real engineers, they can solve quite complex problems, they just cannot read and find the mean of 3.4 million data points in less than an hour in excel.

In very narrow terms a real engineer is someone who knows how to do it fast always the fastest way, do you want the person installing your sewer on your new house using a backhoe or a shovel. The people on this forum fit into that category by default, why else would mecej4 say that the water analysis program would run faster in Fortran -- he was of course correct and a weekend moving the code from C# to Fortran was well worth it.

So I return to my students, I suppose I need to teach them Fortran

John

[Arie_B_]

Maybe you can combine the first two loops into one. If P is rather big, that should save time.

sum=0.0D0

open (unit=2,file="data.txt")

do k=1,P

read(2,*) data(k)

sum=sum+data(k)

end do

u=sum/P

And I have a question. Wouldn't u=sum(data)/P work too? (where sum is the internal function and not a variable

Similarly, after computing u, can we write

VAR=sum((data-u)**2)/(P-1)

and save another loop?

[mecej4]

Arie, your suggested changes create some new conflicts. In general, it is best to do file I/O in big chunks, even the whole file if appropriate, than in little chunks. How the I/O is buffered in the Fortran I/O runtime and the OS will complicate this question.

To write expressions such as sum(data), you have to make sure that the data array has a size of exactly P, and the compiler should know the size of data(). If not, you would have to write sum(data(1:P)).

MKL offers a number of functions to compute standard summary statistics, and it is better to use those functions instead of writing your own code to compute the same statistics.