Use the Schwartz, Luke!
It seems everyone needs to code at least one ELF loader of their own, so here’s mine. Schwartz is a yet another ELF loader and linker that can do a couple of tricks that other linkers can’t do (names not included – any similarity is purely coincidental), like ABI translation. I started it when the gllin binary was released to public in November but never had the time to finish it. It aims to be a generic linker not tied to any architecture or host ABI, but gllin was a good reason to start coding. My next couple of posts will be related to Schwartz as well, so you better be interested!
Schwartz doesn’t use the ELF interpreter mechanism like the ld-linux linker – it compiles to a normal user-space program that needs no special privilege level. Typically the user just runs the linker (the executable name is ld4) passing as a parameter the name of the executable to load and run. Supported architectures are at the moment x86-64, ARM and i386 (the last one untested).
For that to work we have to use some tricks at every level, starting from the loader part. Because every hack has its limits (that make it what we call a hack), if you take The Schwartz code and try to extend it you may hit one of the limits and see that things stop working. There’s nothing inherently unfixable in it but you may need to come up with a new hack.
Its task is loading the contents of an ELF executable into memory at the right locations where the ELF will feel especially comfortable. In other words we construct the memory image of the program out of the image in the executable file. This at first seemed like an easy task because I had zero experience with ELF executables and my last experience with executables was from ms-dos times where all executables were relocatable. So in my endless ignorance I was thinking I’d just reserve a piece of memory, dump the contents there and relocate the code. Obviously this didn’t work because it turns out operating systems stopped using relocatable binaries for normal programs about twenty years ago when I wasn’t paying attention. So to make the program feel at home you have to place the code at the exact addresses it wants.
To run fully in user-space we use a linker script that moves our own code to a non-standard location in the memory image, so that the standard location becomes free and we can load the executable there. Such linker script can be pretty much generated automatically for every platform. Obviously on the target executable could have also used a linker script and chosen an address colliding with our non-standard addresses. In this case the dungeon collapses and we don’t support such executables. The user has to go and modify the script (which is fairly trivial) to be able to run such executable. The user can even go farther and support only a single executable and just link the ld4 with her target program into a single file if she wants to only take advantage of (say) the ABI translation feature for this single program.
By doing that we have both programs in a single memory space / single process, happily coexisting and we gain one interesting feature: If we attach a debugger to the process, we will have the symbols from both executables in place. This means we can load the debug info for either of the programs into the debugger and the debugger will see the symbols in the right places and not get confused. In GDB you can switch the debugged binary in runtime without detaching from the process.
The linker is used only for dynamic executables. It looks at the list of symbols in the external libraries that are used by our target program and resolves each of them by loading the necessary library and finding the symbol. Again we have both programs (ld4 and the target) in a single process so we can share the libraries instead of loading them two times. I use libdl for external symbols rather then resolving them manually but there’s no reason the Schwartz couldn’t recursively load the libraries as well. Currently we support only a very small subset of the defined relocation types but this seems to be more than enough for programs built with binutils (i.e. all programs).
Because we control what we resolve every symbol to, we can override the library symbols with our own when we want. This allows us to play different kinds of tricks on the program.
One such trick is a strace-like tracing of the calls made by the program to library functions. I’ve implemented that for most of the <string.h> calls as an example, this functionality is turned on with the –trick-strace switch.
Another feature is a fake chroot done with simply mangling the path strings passed forward and back between the program and libraries. This is ofcourse not as secure as a real chroot if you allow arbitrary executables, because an executable may use libraries or library functions that we haven’t provided a wrapper for, or use syscalls directly. However, it has the advantage that any user can use it, while normal chroot requires root privileges. This is enabled with –trick-chroot <path>.
Yet another trick could be a user-space implementation of a poor man’s debugger, with the capability to set breakpoints, inspect data, etc., but perhaps not watchpoints (at least not easily) and other fancies. I’m not implementing this.
And yet another trick based on overriding library symbols is C++ exception model translation and ABI translation. More about this in the next post. Look out!