Software Tuning, Performance Optimization & Platform Monitoring
Discussion regarding monitoring and software tuning methodologies, Performance Monitoring Unit (PMU) of Intel microprocessors, and platform updating.
1708 Discussions

Do the terms tlb shootdown and tlb flush refer to the same thing

gostanian__richard
New Contributor I
2,938 Views

One of the Linux tracepoints which perf knows about is tlb:tlb_flush. Is this just a synonym for the more colorful term tlb shootdown.  

0 Kudos
4 Replies
McCalpinJohn
Honored Contributor III
2,936 Views

In Volume 3 of the Intel Architectures SW Developer's Manual (document 325384-071), Section 4.10.5 "Propagation of Paging-Structure Changes to Multiple Processors", there is an explanation:

The process of propagating the changes to a paging-structure entry is commonly referred to as “TLB shoot- down.”

In Volume 2 of the Intel Architectures SW Developer's Manual (document 325383-071), the INVLPG instruction is described as "flushing TLB entries".

So for Intel processors, the "flush" refers to each logical processor invalidating its own TLB entries, while "shootdown" refers to the coordination of these flushes across multiple processors.   

It looks like this is reasonably consistent with the usage in the Linux kernel.

I would be surprised if the meanings are not different in some other processor community.....

0 Kudos
gostanian__richard
New Contributor I
2,936 Views

Thanks John.  From your explanation, it sounds like there can be more flushes than shootdowns. Is that correct? And if so, is there a way to count the number of shootdowns, and more importantly a way to measure the overhead? I don't see a counter or tracepoint referencing shootdowns.

0 Kudos
McCalpinJohn
Honored Contributor III
2,936 Views

TLB management in multiprocessor systems is an active area of research, and I don't know what optimizations are included in various operating systems.  A helpful review of the issues and some of the recent approaches is contained in http://www.cs.yale.edu/homes/abhishek/kumar-asplos18.pdf

The "shootdown" refers to the software coordination of TLB invalidations, so it can be counted by the OS -- e.g., "cat /proc/interrupts | grep TLB" in Linux.

Section 4.10.4 "Invalidation of TLBs and Paging-Structure Caches" in Volume 3 of the SWDM describes the hardware mechanisms that can invalidate TLB entries.  The most common cases are (1) invalidate all non-global TLB entries by writing the CR3 register, and (2) invalidating a single TLB entry when changing a virtual-memory mapping.  I believe that the Linux concept of "shootdown" applies only to the latter case, but that is just my current guess....

Recent processors have hardware performance counters for the underlying low-level events.  E.g., for Xeon Scalable Processors, Event 0xAE ITLB.ITLB_FLUSH counts the number of Instruction TLB flushes (either global or page-specific), and Event 0xBD TLB_FLUSH counts various types of DTLB and STLB flush operations.  It does not look like the HW can distinguish between the different types of events or count the total number of entries invalidated.

The details differ if Process Context Identifiers (PCIDs) are enabled.  This is also described in Volume 3 of the SWDM, but I have not worked through the details using a system that supports PCIDs.

0 Kudos
HadiBrais
New Contributor III
2,936 Views

The term "TLB shootdown" was introduced in the paper titled "Translation Lookaside Buffer Consistency: A Software Approach" in 1989. It is defined in Section 4 of the paper:

Mach pmap modules maintain TLB consistency by forcibly interrupting processors to perform TLB consistency actions (i.e. entry or buffer flushes). This forcible interruption is referred to as “shooting” entries out of remote TLBs, and the entire process of causing remote entries to be invalidated is called a “shootdown”.

Almost all real processors don't support flushing remote TLB entries in the hardware. So TLB shootdowns are usually initiated by the operating system. This is also how Intel defines the term in Section 4.10.5 of the SDM Manual (May 2019).

Sometimes, the term is used to refer to any TLB flush operation whether local or remote. Consider for example the tlb:tlb_flush tracepoint itself. Each time the tracepoint occurs, the number of translations being flushed and the reason for flushing are recorded. In recent versions of the kernel, the following reasons are defined in /include/linux/mm_types.h:

enum tlb_flush_reason {
	TLB_FLUSH_ON_TASK_SWITCH,
	TLB_REMOTE_SHOOTDOWN,
	TLB_LOCAL_SHOOTDOWN,
	TLB_LOCAL_MM_SHOOTDOWN,
	TLB_REMOTE_SEND_IPI,
	NR_TLB_FLUSH_REASONS,
};

Notice how the terms "local shootdown" and "remote shootdowns" are used, which refer to local and remote TLB flushes, respectively.

0 Kudos
Reply