Theo Schlossnagle: Finding leaks on Solaris (w/o Valgrind)

Premise: I write a lot of C code. I run a lot of Solaris.

Sadness: One of my favorite tools ever made is Valgrind. Valgrind does not run on Solaris.

A lot of the C code I write is event-driven and as such (complicated) it is harder to write code and leaked memory is a common residual of this more complicated coding effort. Memory leaks suck. Most of the code I write is systems-level code and as such is exercised heavily and needs to run for months or years without restart. So, leaks more than suck.

Valgrind will tell you exactly where you're leaking (and it feels like it tells you why)... BFM.

On Solaris, we have a different option: libumem.

I compile almost all of my apps against libumem, which has the effect of replacing malloc/free/and friends with a more multi-processor scalable slab-allocator implementation than the default one in libc. A very useful feature in libumem is debugging and leak detection... how?

First, let's assume you've compiled your app with debugging symbols (-g in the compiler flags). Next, I'll assume you didn't have the foresight to link against libumem (-lumem on the linker line). We need to link in libumem and we need to turn on debugging. In the case of this example, our app is the Apache web server httpd.

LD_PRELOAD=libumem.so.1 UMEM_DEBUG=default /opt/apache22/bin/httpd

Now, just find the process ID of exampled (perhaps as easy as a pgrep exampled). It is leaking, so we'll use the process ID 666. To find the leaks (without killing or restarting), we can simply use the mdb umem helper ::findleaks on the running process.

echo "::findleaks -d " | mdb -p 666 > leak-profile.txt

Note you can also do this from a core... very nice.

The output has a summary and then individual profile records grouped by stack trace of allocation, like:

umem_alloc_16384 leak: 203 buffers, 16384 bytes each, 3325952 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
         1d36b60          1d37000   70235c07811a50              197
                           65a468           617900                0
                 libumem.so.1`umem_cache_alloc_debug+0x152
                 libumem.so.1`umem_cache_alloc+0x1a2
                 libumem.so.1`umem_alloc+0xdb
                 libumem.so.1`malloc+0x59
                 libapr.so.0.2.12`allocator_alloc+0x466
                 libapr.so.0.2.12`apr_palloc+0xda
                 bio_filter_out_ctx_new+0x37
                 ssl_io_filter_init+0xb1
                 ssl_init_ssl_connection+0x2cd
                 ssl_hook_pre_connection+0x123
                 ap_run_pre_connection+0xc3
                 ap_process_connection+0x30
                 process_socket+0xa8
                 worker_thread+0x2d8
                 libapr.so.0.2.12`dummy_worker+0x30

mod_ssl... Why are you so mean to me?