EmptyCrate Software. Travel. Stuff.

While debugging my massively multithreaded C++ application I would notice times where the application would seem to pause for a few moments. During one of these pauses I halted the application and attached to it with the debugger (GDB). From within GDB I listed (info threads), switched to (thread <num>) and looked at the stack (bt) of each thread running.

I saw something surprising and very telling. Nearly every single thread that was supposed to be performing work was actually blocked on a mutex inside of either malloc or free.