Investigating mTHP Fragmentation & Inference Latency Spikes in OpenVINO YOLO Workloads

[Sid5]

Hi Everyone,

During an Intel Developer Cloud workshop, I was running object detection inference using YOLOv5 models accelerated with OpenVINO + ONNX Runtime and observed unexpectedly high latency and throughput drops. After some debugging and tracing kernel logs, I identified a memory warning tied to Transparent Huge Page (THP) alignment heuristics introduced in recent Linux kernel changes.

🧠 Root Cause:

• The kernel was aligning anonymous memory allocations ≥2MB to PMD boundaries (per commit efa7df3e3bb5).

• This behavior, while designed to improve THP usage, caused fragmentation when memory chunks were slightly under 2MB (common in model shards or tensor intermediates).

• These misaligned regions couldn’t be coalesced into THPs, leading to degraded TLB locality and increased memory overhead.

️ What I Did:

I submitted a kernel patch that applies THP alignment only to mappings that are a multiple of 2MB, preventing unnecessary fragmentation for variable-sized allocations. This shows many derivatives from the hugging face console as well which I also implemented and fixed as per the errors showcased which can be traced back via cloud logs. This was also integrated with the intel stack using cmake patch build all along the resolution. The fix is under discussion on LKML and has been well received for certain AI workloads. 

Link to kernel discussion:

LKML Thread – mm: limit THP alignment to PMD-sized mappings

Observed Impact (qualitatively):

• YOLO inference throughput dropped significantly with the original THP alignment

• Latency increased due to higher page fault handling and memory fragmentation

• Memory coalescence (via khugepaged) was blocked, impacting model shard loading and reuse performance

Why This Matters for OpenVINO:

• OpenVINO’s inference engine often uses shared memory regions or dynamic model partitions, which may not be PMD-aligned in size

• This leads to poor mTHP usage, especially when tmpfs, shm, or mmap-allocated tensor buffers are used

• mTHP (migratable THP) could benefit from smarter alignment strategies, especially for model server workloads using shared tensor memory (e.g., batching + async inference)

Current Limitation:

Unfortunately, I no longer have access to the Intel Developer Cloud instance I was using for validation, and can’t extract perf or numastat logs at the moment. However, I’m confident the observed performance behavior was directly tied to the memory alignment change — and I'd love help reproducing this in an OpenVINO-native test case.

Has anyone else noticed performance regressions in YOLO (or similar) OpenVINO models due to memory alignment, especially on large batch sizes or async inference? Would appreciate guidance or benchmarks from those who might help validate this further.

I’d be happy to collaborate with the OpenVINO team or community members to dive deeper once benchmarking access is restored.

Thanks & Regards,
Siddhartha Sharma
Intel Software Innovator/Partner

[Wan_Intel]

Hello Sid5,

Thank you for reaching out to us.

Before we proceed with the next steps, I would like to request the following information with you:

• Link of LKML Thread – mm: limit THP alignment to PMD-sized mappings
• Link of the commit efa7df3e3bb5

Once we received the information above, we will reach out to you shortly.

Regards,

Wan

[Sid5]

Hi Wan_Intel,

Thank you for your response.

Please find below the requested references from the attached screenshot as the links produced an error for invalid HTML while posting:

This is the link for the ongoing testing for this patch:

https://lore.kernel.org/all/019401db769f%24961e7e20%24c25b7a60%24@telus.net/

Best regards,
Siddhartha Sharma

[Sid5]

Hi Wan_Intel,

Thank you for your response.

Please find below the requested references from the attached screenshot as the links produced an error for invalid HTML while posting:

LKML Thread – mm: limit THP alignment to PMD-sized mappings.

https://lore.kernel.org/lkml/20231018113455.21723-1-vbabka@suse.cz/

Commit efa7df3e3bb5 – Align anonymous mappings to THP boundaries.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=efa7df3e3bb5b1a5b0c44b77546a8cb9abcfbc37

Let me know if you need further technical details or test case breakdowns — happy to collaborate further.

Best regards,
Siddhartha Sharma

[Wan_Intel]

Hello Sid5,

Thank you for sharing the information with us.

I will escalate this case to relevant team and we will provide an update here as soon as possible.

Regards,

Wan

[Wan_Intel]

Hi Sid5,

Thank you for your patience.

We have received feedback from relevant team. We suggest you submit a pull request via the following link so that you can work directly with our developers:

https://github.com/openvinotoolkit/openvino/pulls

Regards,

Wan

[Sid5]

Hi Wan,

I don't have the write access to this repository you have provided the link for so, can you please grant me the write permission to do so directly?

I am open to generating a fork of this repository then create the pull request as requested.

Looking forward to your reply!

Thanks & Regards,

Siddhartha Sharma

[Sid5]

Hi Wan,

I could not wait for your reply, so I created the pull request from the fork now. Please review for your reference: https://github.com/openvinotoolkit/openvino/pull/31297

In addition, I would like to know about the write access to the repositories status as well and if possible, I would really prefer that way of contributing with new projects.

Best Regards,

Siddhartha Sharma

[Wan_Intel]

Hi Sid5,

Thank you for your information.

rkazants is one of the OpenVINO™ developer, he will review your pull request. You may continue the discussion with him at here.

Regards,

Wan

[Sid5]

Hi Wan,

I see the review now, I have opened the discussion for the same. Please let me know about the direct access to the repository as well for future commits and contributions.

Best Regards,
Siddhartha Sharma

[Wan_Intel]

Hi Sid5,

Thank you for the information.

Thank you for your interest in contributing to the OpenVINO™ project. For your information, direct access to the repository is not available at the moment. To contribute to the OpenVINO™ project, you may refer to the following link:

https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING.md

Regards,

Wan

[Sid5]

Hi Wan,
Thanks for the information, I have opened my pull request already for review. I would like to know how and when the direct access can be made available for Openvino.

Best Regards,
Siddhartha Sharma

[Wan_Intel]

Hi Sid5,

Thank you for your interest in contributing to OpenVINO™.

Let me check with relevant team, and we will provide an update here as soon as possible.

Regards,

Wan

[Wan_Intel]

Hi Sid5,

Thank you for your patience. We have received feedback from relevant team.

For your information, OpenVINO™ repositories are publicly available, and anyone is welcome to clone the latest version for their own use or experimentation.

However, when it comes to code contributions, we kindly ask that users first submit an Issue and engage with the developer community through the Issues section on GitHub. At this time, we do not support direct code submissions (e.g., pull requests) from external contributors, as it may disrupt our internal development flow and quality assurance processes.

We appreciate your interest and engagement, and we hope this guidance helps.

Regards,

Wan

[Sid5]

Hi Wan,
Thanks for the information. I have submitted the PR for review and added some comments as well for further clarification. I will be looking forward to the review process now and work along with the team.

Thanks & Regards
Siddhartha Sharma

[Wan_Intel]

Hi Sid5,

Thank you for the question.

You may continue to work with the developer via the GitHub discussion, and we will proceed with closing this thread. If you need additional information from Intel, please open a new thread.

Regards,

Wan

[Sid5]

Hi Wan,

So my openvino pull request is still pending review from the maintainers for the past week or so. Let me know when they will be addressing the solution here.

Thanks & Regards
Siddhartha Sharma

[Wan_Intel · Accepted solution]

Hi Sid5,

For pull request matters, we recommend to continue your conversation in your pull request thread:
https://github.com/openvinotoolkit/openvino/pull/31297

The maintainers will respond to you accordingly.

This thread has been closed. If you need additional information from Intel, please open a new thread.

Regards,

Wan