Interposing calloc on Linux

The library interpose mechanism offers a nice way to replace some of the standard library functions with our own version for a wide range of purposes (analytics, debug,etc). So how this is happening?

Long story short, you have to create a shared library which will be added to the list of preloaded libraries before the application itself loads. The shared library loader will load your library and any other libraries your application depends on and your application. When symbols are resolved, any symbols will get first resolved from your shared library (if defined there) then from standard library or other shared libraries your application uses. What this means?
Well, if we have a function named malloc in our preloaded shared library, when the application gets loaded by the system, all references to malloc will be resolved to one in the preloaded shared library, instead of one from standard library, so basically, the application will use our version of malloc.
Of course, having it replaced would not help too much if we don’t have a way to call the old (standard library provided) version of malloc when memory needs to be allocated. Fortunatelly, libdl offers a way to do this, in the form of RTLD_NEXT parameter. If you pass this parameter to the function when looking for the address of a symbol (function), it will return (as it says) the next symbol matching the name. In our case the one provided in the standard library.

For example if you create a shared library with the following two functions:

void *malloc( size_t size )
{
	static void * (*func)(size_t);

	printf("Entering %s\n", __FUNCTION__);

	if (!func)
	{
		// get reference to old (libc provided) malloc
		func = (void *(*)(size_t)) dlsym(RTLD_NEXT, "malloc");
	}

	printf("malloc(%d) is called\n", (int)size);

	// call old (libc provided) malloc
	void* ret = func(size);

	return ret;
}

void free( void* pointer )
{
	static void (*func)(void *);

	printf("Entering %s\n", __FUNCTION__);

	if (!func)
	{
		// get reference of old (libc provided) free
		func = (void (*)(void *)) dlsym(RTLD_NEXT, "free");
	}

	printf("free is called\n");

	// call old (libc provided) free
	func(pointer);
}

and add the library to list of preloaded libraries, it will display a list of calls to the malloc and free functions:

$ export LD_LIBRARY_PATH=yourinterposedlibrary.so
$ ls

Entering free
free is called
Entering malloc
malloc(41) is called
Entering free
free is called
....

Using this method, most of the functions from the standard library can be replaced with our own function. However one of them seems to be particulary difficult: calloc.
Why this is more difficult than the other functions? Because turns out that on Linux the dlsym function which has to be used to retrieve the libc version of calloc uses itself this function, which ends up in infinite recursion (since it gets replaced by our own version wich gets called by dlsym which in turn calls calloc which is the interposed version which in turn calls dlsym which in turn … you get the ideea.

In order to fix this issue, seems like we need somehow not to call libc provided calloc until we get reference to libc provided calloc. Easier to say than do .. turns out we need to preallocate some memory wich gets returned when the dlsym function is called to retrieve the address of libc provided calloc. Once that is completed, we can get rid of the hack and use directly the libc provided calloc in order to perform calloc when required, from our interposed calloc function.

Seems complicated or at least not so attractive path to follow. I was wondering what happens if calloc returns NULL until we actually get the libc provided one? It is anyway used in ld.so, the ones who wrote it should take care of handling this situation (maybe failling to load the application itself?). Turns out, if we return NULL for the memory which needs to be allocated by dlsym function while we are trying to get the address of libc provided calloc works just fine (at least on my computer, with my version of Linux). Now things get a lot simpler for interposing calloc:

// our temporary calloc used until we get the address of libc provided
// one in our interposed calloc
static void* temporary_calloc(size_t x, size_t y)
{
	printf("empty calloc called\n");
	return NULL;
}

extern "C" void *calloc( size_t nelements, size_t elemetnSize)
{
	static void * (*calloc_func)(size_t, size_t) = 0;

	printf("Entering %s\n", __FUNCTION__);

	if (!calloc_func)
	{
		// before trying to get the libc provided
		// calloc, set the calloc function to the temporary
		// one returning NULL.
		calloc_func = temporary_calloc;

		// dlsym will call again this calloc in which we are,
		// but seems like it handles just fine if we are
		// returning NULL
		calloc_func = (void *(*)(size_t, size_t)) dlsym(RTLD_NEXT, "calloc");
	}

	void* ret = calloc_func(nelements, elemetnSize);

	return ret;
}

Adding this to the previous shared library and interposing it, will also display when the application calls calloc proving that interpose was successfull.

  1. Hey man,

    I just wanted to thank you about your solution. I’ve been searching soooo long for this crap. I had a solution of my own (essentially mmaping my own page and creating a fake allocator for the first callocs) but yours is much much more elegant (and faster!).

    Thanks again
    Nick
    http://www.securitee.org

  2. Hey, you’re welcome. I’m glad it helped you :mrgreen:

  3. Your post saved me from a very nasty situation 😀
    Thanks!
    I am working on a tool to detect overflows, for an university project.

  4. I’m glad it was helpful :mrgreen:

  5. The reason this works is becaus the dlsym code calls realloc when the calloc call returns NULL.
    If you try to write some code that replaces calloc, malloc, realloc, and free, you wind up with the same problem.

Leave a Comment