P-State invariant TSC on Nehalem platforms with multi-packages
The following comment was made by Intel:
"The time-stamp counter on Nehalem is reset to zero each time the
processor package has RESET asserted. From that point onwards the TSC
will continue to tick constantly across frequency changes, turbo mode
and ACPI C-states. All parts that see RESET synchronously will have
their TSC's completely synchronized. This synchronous distribution of
RESET is required for all sockets connected to a single PCH. For large,
multi-node systems, RESET might not be synchronous."
The problem with this statement is that based on Intel's own SMP architecture, when a system boots, there is one BIOS designated boot service processor (BSP) that the kernel is loaded on. It is then the kernels job to bring online the application processors (APs) at the appropriate time to enable SMP.
How can one do a RESET synchronously across all cores (which might be in a different packages but even within the same package) to ensure the TSCs are indeed synch'ed?
i.e. can you have a sync'ed TSC across different packages or are the CPUID bits only a per package basis? If they are on a per package basis, again given the BSP/AP boot process, how is a synchronous TSC achieved?
Within a single package, there should be only a single TSC counter. Unless you are writing a BIOS, you don't normally have control over the method for synchronizing TSC among packages. Presumably, the BIOS would do its best either to send a synchronous reset or to initialize all the counters on a shared memory system as close as possible to the same value. As far as I know, BIOS writers' guides are not publicly released.
No. There is a TSC counter per CPU. I want to know if the TSC even within the same package of cores will all drift at the same rate? Also what are the conditions that cores will be out of sync provided the TSC is P-state invariant?
I am not an expert on TSC synchronization but I do know a bit about system initialization process.
1. All cores in a single package observe the same /RESET line. Thus, all cores will be reset at the same time when they observe the /RESET signal, regardless of the fact that only one will continue the boot process.
2. Same /RESET line is visible to all CPU packages that are installed on the same mainboard meaning that all those packages will observe the /RESET signal at the same time.
In short, all CPUs execute INIT at the same time. The CPU core that asserts the bus first will be designated as BP and continue executing the code while all other cores will be designated as APs and enter into wait-for-SIPI state. After BP performs initial system configuration, it is then a task for the operating system to initialize APs by sending proper SIPI (Startup IPI) APIC message(s). As far as I know it is possible to bring up the APs without resetting them again so the TSCs stay synchronized.
For more details check APIC programming and MPS specification.