Bad news, I’m gonna talk about OABI again. I just want to write down what I found about it before I forget, so that it gets indexed and a person needing to know something can find it on google.
I was told by a gcc hacker that it was based on the APCS32 ABI whose specification can be found here. The specification is however very vague about some parts, and other parts are simply different from OABI, so I’ll point these out and refer to APCS32 in other places, and compare with EABI also.
Control arrival. One thing that is not specified at all in either APCS32 or EABI is the program entry point requirements. These may be system-specific but the Linux Standard Base has no mention of ARM entry point either. The only reference thus is the Linux kernel code. Qemu-arm code is based on it. The requirements don’t seem to have changed between OABI and EABI and they’re also pretty much identical to the x86 entry point requirements which can be found in the SysVr4 docs, modulo some of the tags put on the stack before entry. They can be found in Linux or qemu and I’m not gonna list them here.
APCS Variants. The APCS32 document specifies 16 incompatible variants based on four different properties that can have two possible values each. Linux OABI is the 32-bit case (as opposed to 26), with implicit stack-limit checking (as opposed to done in software), floating-point arguments/return values passed in registers and on stack (i.e. FPU registers are not used for that) and is non-reentrant (except libraries).
Arguments passing. Register have the same meanings as in APCS32 with first four words of argument list passed in registers and the rest on stack, with the possibility of a single argument split between the two.
Floating-point values. There’s no mention of their encoding in APCS32 but it seems to be the standard IEEE 754 encoding – with a small caveat… Doubles and long doubles have their first 32-bit word swapped with the second word, when compared to EABI or x86. The same applies to both of the individual doubles inside a double _Complex and inside long double _Complex.
Return values. Here we have the same three variants as in APCS32: no return value, return value in register(s) and return value in an implicit pointer passed as arg0. There is a very tricky difference from APCS32 though: when is the second variant chosen and when is it the third one. APCS32 recognises something it calls simple types which it defines as anything that fits in four bytes. Anything bigger is returned through a pointer. In OABI there seems to be a similar idea except the simple type is defined differently: All C basic types are simple, even if they exceed the word width. In addition to this a struct seems to be considered simple as long as it has only a single member whose type is simple (possibly a struct) and not larger than one word. Arrays are never simple and unions are simple if all their fields are simple.
The $100 question is how do you return an object of a simple type wider than one word in a register? APCS32 allows only r0 to be used for that, but gcc doesn’t mind using also r1, r2 and r3. So a long long int, double, long double, int _Complex or a float _Complex will all be returned in the r0-r1 pair, while double _Complex and long double _Complex get returned in r0-r3.
Alignment. Pointers have to be word-aligned only and this applies also to the stack pointer on call. This is nothing special but if you want to mix OABI and EABI it becomes a major caveat because EABI requires the stack to be aligned on 8-bytes in inter-linking-unit calls. If you forget about it and call and an EABI function from OABI context you will get the strangest and extremely hard to debug results, such as glibc sprintf() returning a wrong value, which can very painful.
Another change that happened at the same time as the OABI to EABI switch in Linux was a switch from setjmp/longjmp based C++ exceptions (the generic, cross-platform way) to a new, faster model (EABI does specify how exceptions should be handled and how stack unwinding works, while APCS doesn’t – this aspect is known as C++ personality across the docs and code). I am not describing it here.
If something of the above is wrong, please lemme know.