CPU time vs. time

[NotThatItMatters]

I have a time simulation application running a fairly large model which requires a 64-bit executable to run.  Without the 64-bit executable, I get virtual memory errors.  Running the model in question, I am noting that, at this point, the model has been running 6 days and yet the CPU time is only 42,000 seconds.  With smaller models and runs, CPU time and real time are pretty much synchronized.  What sort of thrashing is going on which is causing this discrepancy?

[Bernard]

Jim,

Xperf tool can monitor pagefault activity and I suppose there is also an option to get the exact code location(presumably inside function) which causes page faults.

[SergeyKostrov]

This is a short follow up. Here is a screenshot that demonstrates CPU Usage and Page File Usage when there are lots of I/O operations during some processing: As you ca see CPU is Not busy.

[Bernard]

Task manager has low "granularity" of cpu load measurement.You can see that there is low percentage of cpu time spent servicing those I/O, but from user mode code "point of view" such a operations are very costly.

[SergeyKostrov]

>>...Task manager has low "granularity" of cpu load measurement... Windows Task Manager has four selections for Update Speed: Paused, Low, Normal and High Last three always show right CPU Usage and when I/O operations are completed CPU Usage increases to 99% on a one CPU system. I could post a scrrenshot which demonstrates it.

[NotThatItMatters]

Thank you all for all your insight.  I am busy taking the code apart and purging it of unnecessary arrays.  This is going to take some time, being as there are more than 10 linear solver routines compiled in the "ancient" code but really only 3 active ones and actually just one appropriate set of memory needed for all solvers.  I am trying to free up: 1) stack and 2) allocatable arrays.  I am noting that simply increasing memory to 8 Gb now makes the original model run with CPU time = clock time, although a simple look at memory requirements would indicate it is doing some page swapping to utilize VM.  This is showing me the need for appropriate allocation and maintenance of arrays.

[Bernard]

I mean timer resolution which is around ~10ms.Clockres tool will report actual timer resolution.

[SergeyKostrov]

>>... I am noting that simply increasing memory to 8 Gb now makes the original model run with CPU time = clock time... Do you mean Physical or Virtual memory?

[NotThatItMatters]

Physical memory.  Virtual memory has never been a problem.  The machines in question have a great deal of disk space and a large allotment of virtual memory (32 Gb if memory serves).

[John_Campbell]

NotThatItMatters,

It is good to find that you have obtained improved performance, at least increased %CPU. This is probably due to the increase in physical memory.
The other change was changing the order of array index usage to improve sequential memory access, as you identified with A(KK,3,3) to A(3,3,KK). If processing with DO k; DO j; DO i, this should improve the availability of data in the cache, as Jim has identified in a number of posts. This might not change the %CPU (once page faults are corrected) but will significantly improve (cache) performance and run times.
The combination of both of these can be very effective.

There are two further levels of improvement; vectorisation and OpenMP. Vectorisation is easy as it is just a compiler option, again assisted by sequential memory usage, while parallelism will probably result in more posts on this forum.

Hopefully you have achieved a significant improvement,

John

[Bernard]

>>>Windows Task Manager has four selections for Update Speed:>>>

I meant binding of measurement to specific threads/interrupts.This functionality is not supported on task manager.

[SergeyKostrov]

>>>>Windows Task Manager has four selections for Update Speed... >> >>I meant binding of measurement to specific threads/interrupts.This functionality is not supported on task manager.... Task Manager was not designed to provide all that information. I simply don't understand how it could change matters in the case. It was already informed that additional Physical memory helped and, I think, we reached "Bottom of the Ocean" in that discussion.

[Bernard]

>>> I think, we reached "Bottom of the Ocean" in that discussion.>>>

Agree with you.

[jimdempseyatthecove]

>> I think, we reached "Bottom of the Ocean" in that discussion.

Only if all platforms the app runs on can install at least 8GB of RAM. If this is not the case then the programmer must take into consideration the paging aspects of a virtual memory system. Many larger than physical RAM problems can be quite nicely solved using Virtual Memory and Paging. It is unfortunate that too many of the younger programmers do not grasp this concept. This apparently was an old program to begin with. My guess is this old program ran quite nicely at one time on a system with a few MB and using sequential files. I am not suggesting that you revert back to sequential files, rather that the programmer think of the process in sequential terms (and how to get the most out of each sequence within the physical memory constraints of the system).

Jim Dempsey

[jimdempseyatthecove]

FWIW - Many of these "legacy FORTRAN" programs were written to run on DEC KA10 (256 K words), DEC KI10/KL10 (8MB), DEC VAX-11/780 (max RAM 8MB), DEC VAX 7000/10000 (~3.5GB RAM). In the earlyer days, large problems may have been 100x the size of physical RAM, and input/output was from/to magnetic tape. In those days, the programmer paid attention to the sequence of operations in the program to minimize the seek and number of reads/writes. The art in programming on these machines (read "low memory"/"less memory" than required) is lost for the current generation of programmers. A really good programmer will recognize situations when "more" is "less". A good example of this might be a matrix multiplication or FFT were:

Method A: fastest when run with sufficient RAM
Method B: slowest when run with sufficient RAM

Then when a problem larger than physical RAM is presented to the methods, Method B becomes far superior. The general problem (IMHO) CS courses generally do not teach students when to choose method B (more is less), nor do they discuss method B (other than as an example of what not to use).

Jim Dempsey

[Bernard]

That is true.Sadly todays young generation of programmers is working at very high level of abstraction with scarce knowledge of the low level details.

[NotThatItMatters]

I am currently examining the written code for ways of minimizing allocation.  Let me explain the problem: a 2x2 block matrix needs to be solved.  The main block (item 1, 1) is quite large but very banded.  Hence its approximate solution is quite simple using standard techniques.  The 1, 2 and 2, 1 entries are the main memory hogs.  On input, these matrices are quite (emphasis here) sparse.  However, the banded matrix solution of 1, 1 yields a dense working matrix for 1, 2.  All of these matrices (1, 2  2, 1  2, 2) along with their working counterparts are currently allocated as full matrices in the "main" and passed through several subroutines to fill and solve them.  I am trying to modularize the initialization and use of these arrays.  This is not a trivial matter, although several of you probably have insight in solving similar problems.

[jimdempseyatthecove]

As you go through the code, look at the subroutines to see if they can work on sections, or possibly, individual rows and columns, in a chunk by chunk bases. Current (high-paging) code

sub1(A,B)
do iRow=1,nRows
do iCol=1,nCols
doWork
end do
end do
end sub1

sub2(A,B)
do iRow=1,nRows
do iCol=1,nCols
doWork
end do
end do
end sub1
...
call sub1(A,B)
call sub2(A,B)
...

You might find it more benificial to use

sub1(A,B,iRow)
do iCol=1,nCols
doWork
end do
end sub1

sub2(A,B,iRow)
do iCol=1,nCols
doWork
end do
end sub1
...
do iRow = 1,nRows
call sub1(A,B,iRow)
call sub2(A,B,iRow)
...
end do

The above is tile by row but you could tile by square-ish sections of the matrix too. The idea here is to make use of the data as much as possible should paging be required.

Jim Dempsey