tbbmalloc_proxy: does not replace free() inside Docker container

Milton_Y_Intel · ‎08-01-2017

Hi,

I use tbbmalloc_proxy library in application to replace all malloc and free w/ tbbmalloc's equivalent. The application crashes when run inside a Centos 6 or Ubuntu 16.04 Docker container.

The crash is:

*** glibc detected *** myapp: munmap_chunk(): invalid pointer: 0x00007f0b067d5920 ***

The application runs fine when it's not running inside a Docker container.

The crash occurs in a call to udev's udev_enumerate_get_list_entry() API. The udev call is initiated from FlexLM API. The reason for the crash appears to be a free() not getting replaced with scalable_free(). This leads to libc's free() trying to free memory that was allocated a malloc() that was performed by scallable_malloc().

Memory was allocated here:

73 Hardware access (read/write) watchpoint 3: *(0x2aaaf2c9c260)
 74 
 75 Value = -221658432
 76 0x00002aaad11a0274 in rml::internal::internalPoolMalloc(rml::internal::MemoryPool*, unsigned long) ()
 77    from .../linux64/libtbbmalloc.so.2
 78 (gdb) where
 79 #0  0x00002aaad11a0274 in rml::internal::internalPoolMalloc(rml::internal::MemoryPool*, unsigned long) ()
 80    from .../linux64/libtbbmalloc.so.2
 81 #1  0x00002aaad11a0e76 in scalable_malloc () from .../linux64/libtbbmalloc.so.2
 82 #2  0x00002aaad484f409 in malloc () from .../linux64/libtbbmalloc_proxy.so.2
 83 #3  0x00002aaad5482022 in __strdup (s=0x2aaaf4c9d5a0 "/sys/devices/virtual/block/dm-0") at strdup.c:43
 84 #4  0x00002aaaf50dcb55 in syspath_add (udev_enumerate=0x609100, syspath=0x2aaaf4c9d5a0 "/sys/devices/virtual/block/dm-0")
 85     at libudev/libudev-enumerate.c:164
 86 #5  0x00002aaaf50dd91d in udev_enumerate_get_list_entry (udev_enumerate=0x609100) at libudev/libudev-enumerate.c:273
 8

Memory is freed here:

#0  __libc_free (mem=0x2aaaf2c9c260) at malloc.c:3705
#1  0x00002aaaf50dd9b1 in udev_enumerate_get_list_entry (udev_enumerate=0x609100) at libudev/libudev-enumerate.c:314
#2  0x00002aaacc3a6c05 in ?? () from .../linux64/myapp.so
#3

Here's the udev code. malloc happens on line#273 and is freed in line# 314.

246     struct udev_list_entry *udev_enumerate_get_list_entry(struct udev_enumerate *udev_enumerate)
247     {
248             if (udev_enumerate == NULL)
249                     return NULL;
250             if (!udev_enumerate->devices_uptodate) {
251                     unsigned int i;
252                     int move_later = -1;
253                     unsigned int max;
254                     struct syspath *prev = NULL;
255
256                     udev_list_cleanup_entries(udev_enumerate->udev, &udev_enumerate->devices_list);
257                     qsort(udev_enumerate->devices, udev_enumerate->devices_cur, sizeof(struct syspath), syspath_cmp);
258
259                     max = udev_enumerate->devices_cur;
260                     for (i = 0; i < max; i++) {
261                             struct syspath *entry = &udev_enumerate->devices;
262                             size_t move_later_prefix = 0;
263
264                             /* skip duplicated entries */
265                             if (prev != NULL &&
266                                 entry->len == prev->len &&
267                                 memcmp(entry->syspath, prev->syspath, entry->len) == 0)
268                                     continue;
269                             prev = entry;
270
271                             /* skip to be delayed devices, and add them to the end of the list */
272                             if (devices_delay_end(udev_enumerate->udev, entry->syspath)) {
273                                     syspath_add(udev_enumerate, entry->syspath);
274                                     /* need to update prev here for the case realloc() gives
275                                        a different address */
276                                     prev = &udev_enumerate->devices;
277                                     continue;
278                             }
279
280                             /* skip to be delayed devices, and move the to
281                              * the point where the prefix changes. We can
282                              * only move one item at a time. */
283                             if (move_later < 0) {
284                                     move_later_prefix = devices_delay_later(udev_enumerate->udev, entry->syspath);
285
286                                     if (move_later_prefix > 0) {
287                                             move_later = i;
288                                             continue;
289                                     }
290                             }
291
292                             if ((move_later >=0) &&
293                                 strncmp(entry->syspath, udev_enumerate->devices[move_later].syspath, move_later_prefix) != 0) {
294
295                                     udev_list_entry_add(udev_enumerate->udev, &udev_enumerate->devices_list,
296                                                 udev_enumerate->devices[move_later].syspath, NULL, 0, 0);
297                                     move_later = -1;
298                             }
299
300                             udev_list_entry_add(udev_enumerate->udev, &udev_enumerate->devices_list,
301                                                 entry->syspath, NULL, 0, 0);
302                     }
303
304                     if (move_later>=0)
305                             udev_list_entry_add(udev_enumerate->udev, &udev_enumerate->devices_list,
306                                                 udev_enumerate->devices[move_later].syspath, NULL, 0, 0);
307
308                     /* add and cleanup delayed devices from end of list */
309                     for (i = max; i < udev_enumerate->devices_cur; i++) {
310                             struct syspath *entry = &udev_enumerate->devices;
311
312                             udev_list_entry_add(udev_enumerate->udev, &udev_enumerate->devices_list,
313                                                 entry->syspath, NULL, 0, 0);
314                             free(entry->syspath);

Has anyone had experienced w/ this issue or have any suggestions? Are there limits on tbbmalloc_proxy.so when running inside a Docker container?

Thanks,

Milton

Alexei_K_Intel · ‎08-07-2017

Hi Milton,

Unfortunately, I am not familiar with Docker and never tried to use libtbbmalloc_proxy.so inside it. Can it be so that application is linked with static C-runtime? Do you observe similar behavior with simplified examples?

Regards,
Alex

Vladimir_P_1234567890 · ‎08-11-2017

Hello,

did you set-up LD_PRELOAD for proxy when you configured docker for your app?

--Vladimir