scalable_malloc fragmentation problem

[zachsaw]

Hi,

Is the latest version v3.0 update 2 commaligned release suppose to use about the same amount of memory as the crt malloc (which calls heapalloc)?

I'm still seeing very bad fragmentation problem with scalable_malloc / scalable_free - the same app when using malloc ends up with a peak memory (VM Size in Task Manager of XP) of 11MB but with scalable_malloc, it kept creeping up non-stop (I tested it up to 110MB before deciding it's not going to stop and killed it). TBB's allocator is being used in as a garbage collector's memory allocator - multiple garbage collector threads are responsible for deleting the garbage objects (which ends up calling scalable_free).

I understand the older version never released any memory back to the OS but the later versions seem to have changed that behavior. I do see this growing allocation of VM without releasing a bit of a problem for 32-bit systems where the 2GB boundary isn't very hard to blow out.

Edit:

Additional info on the usage pattern - plenty of 32KB allocations that are not released until the GC kicks in. The GC is generational which means some of the 32KB allocations would not get released if it managed to get into the older generation when it survived a gen1 collection or when it's still being held by a reference. I can see this as a big problem for any allocators but winheap has no problems keeping the fragmentation low.

[Krishna_R_Intel]

Hi,
This is interesting. Seems like there's a memory leak at first glance. Do you have a simple, shareable test case for us to verify if this is a potential bug or not?

Meanwhile, I thought I'll point you to a paper that has design discussions about TBB's scalable memory allocator which is accessible from the following link: http://www.intel.com/technology/itj/2007/v11i4/5-foundations/1-abstract.htm

I will continue investigating this problem and shall post any updates on the problem to this thread.

Thank you for using the forum.

Best Regards,
Krishna

[zachsaw]

It's quite hard to write a test case to reproduce the problem seeing that the usage pattern is really complicated and would require a very carefully hand-crafted test case which is highly time-consuming with no guarantee of results. The only benchmark we had was Windows heapalloc, which released memory back to the OS with due diligence.

The issue is touch and go too - minor changes in our code would cause the problem to disappear.

[Alexey-Kukanov]

It's hard to say what might cause excessive memory overhead without having a test case or at least a trace of allocations. For "big objects", the allocator uses some heuristics to define for how long to keep memory in internal pools before fully releasing. Might be these heuristics don't work well for your scenario.