CPU time vs. time

[NotThatItMatters]

I have a time simulation application running a fairly large model which requires a 64-bit executable to run.  Without the 64-bit executable, I get virtual memory errors.  Running the model in question, I am noting that, at this point, the model has been running 6 days and yet the CPU time is only 42,000 seconds.  With smaller models and runs, CPU time and real time are pretty much synchronized.  What sort of thrashing is going on which is causing this discrepancy?

[Bernard]

What do you mean by real-time?

I suppose that CPU time is the total  activity of your process's thread(s).In that time interval(6 days) threads will not be running whole the time.They will be wapped out and swapped in so the total time accumulated can not be exactly equal to 6 days.For more precise measurement can you use Xperf?

[SergeyKostrov]

>>...I have a time simulation application running a fairly large model which requires a 64-bit executable to run. Without the 64-bit >>executable, I get virtual memory errors... 32-bit applications have limitations on how much memory could be allocated. For example, on a Windows platform the limit is 2GB. Also, for tests with 32-bit applications check Virtual Memory ( VM ) settings on your platform ( it is Not clear what platform you're using ) and increase values for Initial and Max sizes of VM. Let's say you're using CPU_TIME function. On a system with many cores a Fortran function CPU_TIME calculates time for all CPUs used during processing. For example, if processing is done in 30 seconds on a system with 8 logical CPUs and all CPUs were used ( processing was done in parallel on 8 threads ) then CPU_TIME will return 240 seconds ( 8 x 30 seconds ). If the same processing is done on 1 CPU ( 1 thread ) then CPU_TIME could return a value greater than 240 seconds.

[SergeyKostrov]

>>...For more precise measurement can you use Xperf?.. How is it going to help?

[Bernard]

Was @NotThatMatters talking about some fortran timing function CPU_TIME?

[NotThatItMatters]

The 64-bit application is using 25% of the CPU all the time.  This is from task manager.  However, as noted in the first post, it is connected for 7 days now and has run CPU_TIME = 49000 seconds.  In a "normal" run, meaning far smaller memory footprint, the disparity between clock time and CPU time is negligible.  A run which takes 49000 seconds can be left at the close of the day and on return the next morning, it will be complete.

[Bernard]

I think that at OS-hardware level such a function like CPU_TIME relies on real clock timer exactly as Xperf.Here I mean Windows platform.On Linux it will also query real time clock but the underlying OS dependent implementation could be different.

[Bernard]

Are you interested in total accumulated time in seconds?

[NotThatItMatters]

We always post "elapsed time" when running our app.  Typically elapsed time and UTC are comparable.  Let me just point out that I gave you some bad information.  The simulation is running on a Windows 8 machine and I was unfamiliar with the "new" task manager.  According to what I am now reading, I am seeing memory being eaten at nearly 100% by the app, but CPU is being used in small increments (0.4%, for example), and the "Disk" is being used 50%.  I am not certain what this all means.

[Bernard]

Regarding memory consumption is it total process working set ? In one of my earlier posts I advised to do not rely on task manager percentage of time measurement.Because it is based on real time clock interval and sometimes your thread will run shorter than the timer interval.Better option is to mesure cpu cycles charged to your thread.It can be done by process explorer.Small increments are probably related to deficiancy of timer interval because of lower time resolution.The same is related to disk I/O percentage measurement.Better option is to look at number of I/O operation per second.Small increments are related to activity of your applications where percentage of time when cpu spent on running your app is growing as staircase function.

[Bernard]

Is CPU_TIME exported by fortran.dll?

[NotThatItMatters]

The app is a compiled exe.  CPU_TIME is the generic intrinsic subroutine called from within the code to report processor time.

[SergeyKostrov]

>>...Running the model in question, I am noting that, at this point, the model has been running 6 days and yet the CPU time is >>only 42,000 seconds. With smaller models and runs, CPU time and real time are pretty much synchronized. What sort of >>thrashing is going on which is causing this discrepancy? 1. You're mixing CPU time with I/O time (!). 2. If your application is very busy with reading from and writing to a model file, especially when model is too big and Virtual Memory ( VM ) is used, then CPU doesn't do anything for your application and it simply waits for a completion of these I/O operations. 3. Actually, processing of your model was done in 11.67 hours ( 42000 / 60 / 60 = ~11.67 ) and all the rest time was spent on I/O operations ( or, VM operations ). 4. Take a look at Performance property page of Windows Task Manager and you will see lots of "pillars". Please make a screenshot and post it for review. 5. Let me know f you need an example ( a screenshot of processing of some my algorithm with heavy load of I/Os, that is VM operations ) of these "pillars" ( I could easily provide it ). 6. What system do you use? How much Physical Memory is installed? What are settings for Virtual Memory? What are sizes for your models? >>...With smaller models and runs, CPU time and real time are pretty much synchronized... 7. Because it uses Physical Memory and it doesn't use too much Virtual Memory. Again, use Performance property page of Windows Task Manager to verify.

[John_Campbell]

Do you have an expectation of how much memory your simulation requires? Is it exceeding the physical memory installed, after allowing for the memory requirement of the operating system and other processes?
There is a suspicion that you are loosing time due to I/O delays, hardware conflicts or virtual memory pageing.
In Task Manager, include "Page Faults" and "PF Delta", as these indicate paging taking place with your process/simulation.
Also "Working Set (Memory)" and "Peak Working Set (Memory)", in comparison to installed memory is another indicator.
The easy solution for this delay cause is to have sufficient physical memory installed and also check that your Vitrual memory configuration is adequate. ( on Win 7 see Control Panel > System > Advanced System Settings >  Performance Settings > Advanced > Virtual Memory )
Memory is cheap and unless your new memory demand is over the top ( from your 32-bit past) this might solve your problem.

Alternatively there could be hardware clashes with other processes for I/O, such as virus checkers etc, although these should not be significant. How many simulation processes are you running ?

Task manager might not be the best measure of performance, but it should be able to identify the most likely cause.

John

[Bernard]

Hi NotThatItMatters,

what do you mean by saying real-time because it has ambiguos meaning?

[Bernard]

As I pointed it out in my first post your code cannot fully utilize your cpu resources unless system kernel mode threads are spawn and run at elevated IRQL level = 0x2 or your thread(s) is operating at real-timepriviledge level but it is not recommended because of induced starvation which can happen to some system thread.Your application must share cpu with hundreds of threads many of them running more priviledged.Moreover on multicore system code of CPU_TIME can continue running on one core when for example part of your code can be swapped out so you will not be really measuring performance of your code.For accurate performance measurement task manager is not recommended.You can profile your application with Xperf toolkit which can provide you with the nice graphical  breakdown of thread activity both operating in user mode and kernel mode.

[SergeyKostrov]

>>...As I pointed it out in my first post your code cannot fully utilize your cpu resources unless system kernel mode threads are >>spawn and run at elevated IRQL level = 0x2 or your thread(s) is operating at real-time priviledge level The problem is Not related to real-time processing and the expression has to be changed to total-time of processing.

[Bernard]

Memory  load and store operations(Port2 and Port3) can be performed in parallel with heavy floating point calculations.The main problem will be waiting on I/O completion(asynchronous waiting) and when you inspect your thread's call stack you will see that Kernel context switching functions are on top of stack.

As John pointed out please check for number of page fault operations occuring per second because resolving page fault is very costly and can slow down your system(significant overhead created by interrupting cpu calling default kernel mode page handler(located in IDT) and calling disk.sys driver).Moreover some AV can interfere with I/O operations starting from hooking ReadFile and WriteFile to installing filter drivers and checking the buffers beign sent and received.

[Bernard]

I meant thread's priviledge level,not an application running in real-time(in domain of video simulations).

[SergeyKostrov]

>>...Memory load and store operations( Port2 and Port3 ) can be performed in parallel... >>...Kernel context switching functions... >>...cpu calling default kernel mode page handler... >>...located in IDT... >>...disk.sys driver... >>...installing filter drivers... Please be advised that this is a forum for Fortran developers and this Not a forum Low Level driver software developers. Please wait for a response from NotThatItMatters and don't clutter the forum with low level hardware details.

[Bernard]

And how my answer differs from for example John's answer he also used technical jargon?