In Part 1 of this series, we proceeded from a simple, console-based “Hello, World” to a gui-based version where we directly call Win32 functions from NASM. In that section, we postponed discussing a few important things, like calling convention. We will look at that in depth in this second part.
C calling ASM, Please Respond
Let’s jump right in and do a very small task here: we’ll write a function that adds two integers in assembly, and call that function in C. I’m going to create 2 files for this exercise, asum.asm and sum.c.
Here is my assembly code:
Things to note:
- We preserve ebp by pushing it into the stack. This is because we want to use ebp as our stack reference. After pushing ebp onto the stack, we then move esp to ebp, creating a stack frame.
- There is a way that C calls functions, known as C calling convention. In this convention, the caller pushes arguments onto the stack in a right-to-left manner. Therefore, the first argument to be passed to the function is the topmost on stack. This is useful in creating variable-argument list functions, like printf. As soon as the callee reads the first argument, it will have to know how many more arguments to expect. In this case, therefore, the C caller pushed the integers b and a (in that order) to the stack. On top of that, we pushed the old value of ebp onto the stack as well. So now the stack would look like this:
- Remember that the stack grows from high memory to low memory. Therefore, [ebp] now points to where we saved the old value of ebp (which is 8 bytes), [ebp+4] is the return address, pushed implicitly by the call instruction that called us, [ebp+8] is a 4-byte integer which is the first argument a, and [ebp+12] which too is a 4-byte integer, the second argument b.
- What we now do is pretty straightforward: we move the first argument to the accumulator, eax, add the second argument to eax. Now, eax has the sum of the two integers. We will leave it there because that’s where the calling C function expects to find it.
- Finally, we pop the old value of ebp from the top of stack. Since the number of pushes equal the number of pops, there are no stack adjustments to be made.
- Note that we didn’t clear the stack. This is again as per C calling convention, where clearing the stack is done by the caller, not callee.
The C code calling this function is pretty straightforward:
Notice that even though the assembly function is named “_asum“, in C, we define it to be “asum“. Why so? Because the C compiler will mangle, or decorate the function name with an underscore prefixed.
We then proceed to assemble, link and run, like so:
Note that we chose elf as the format of the compiled assembly. This is because we want to keep the function name to “_asum” which then, can be called from C as simply “asum“.
ASM Calling C, Please Respond
Now that we had a look at how a C-callable assembly function could be written, the next logical step is to check if we can do the same vice versa: to call a C routine from within assembly.
In fact, if you look at Part 1 of this series, you might notice that the call to printf was exactly like that. Nevertheless, we will look into another example here.
In case you’re wondering, “Ayubowan” is the traditional welcome in Sinhalese language (which, if you’re curious, has a literal meaning of ‘may you live longer’).
I want you to note this statement: add esp, 4 . Why does the caller have to raise the stack pointer by 4 bytes?
Remember that puts accepts a pointer to a char array. That pointer in 32 bits is 4 bytes long. We, the caller, pushed that 4-byte integer value to the stack, to be picked up by puts. Now, once puts is done processing, we want to clear the stack by raising the esp (thus effectively clearing the stack).
Assembling and running is straightforward:
Remember that this time, the driver program (or the program that has the main entry point) is assembly; this is why we had to run nasm with -fwin32. (In the previous example, we ran nasm with -felf and gcc with -m32.)
Calling conventions decide three important things:
- What happens to the stack, before and after a function call?
- What happens to the registers, before and after a function call?
- How is the function name decorated?
There are lots of calling conventions: __stdcall (Win32 default)and __cdecl (C/C++ default), in addition to a whole lot of others like __fastcall, __clrcall, __thiscall and __vectorcall (plus some obsolete ones like __pascal, __fortran and __syscall).
Now, before we go down to the all the implementations, let us quickly list down a handy shortcut table of the two most important conventions so that we all remember who does what.
|Argument passing convention||By value, unless a pointer / reference type is passed||By value, unless a pointer / reference type is passed|
|Who cleans up the stack||caller||callee|
|Name decoration||Prefix underscore||Prefix underscore, then @ sign, then the number of bytes to pop from stack|
Now, the next task is for us to write an assembly function that gets called successfully from C, using any of these conventions, that adds 3 numbers and return the sum (to the caller). We will first look at __cdecl, and then __stdcall. After that, it is a minor change from __stdcall to __fastcall. In each case, a C main() method will call our assembly function using one of the 3 conventions.
When writing the C function, we come across one of those annoying differences between compilers. Imagine the __fastcall version: in Microsoft C comipler, the function prototype has to be something like this.
Whereas in GCC, it will have to be something like this:
The difference has to do with how each compiler implements the calling conventions.
Luckily, we can use the preprocessor to ease our burden here a lot. Although not very pretty, this simple macro will certainly save the trouble.
I owe the answer to this stackoverflow question.
Now that we have a way to correctly mark our function prototype with the calling convention, writing the calling function is trivial:
The real task is now, writing the 3 routines csum, ssum and fsum in NASM, that honor these 3 calling conventions.
Sum via __cdecl–csum.asm
First up, is the easiest one: the cdecl version.
As seen here, the callee does not pop any parameters from stack, except the ebp it pushed for its own usage. The arguments from right-to-left are found at [ebp+12], [ebp+8] and [ebp+4], respectively. (Remember that ebp itself contains the return address.) The return value is left in eax, where the C routine finds it.
Sum via __stdcall–ssum.asm
In __stdcall, all arguments are pushed to the stack just the same way as in __cdecl, but this time, it is the callee’s responsibility to clean the stack. This is achieved by the operand to the ret opcode, which tells the number of bytes to pop from the stack.
Plus, the name decoration is: prefix underscore, function name, @ sign, and the the length of the argument list, in bytes. If the function takes no arguments (i.e. void), this should be 0.
Also, note that the return value is left in eax, just like in __cdecl.
Sum via __fastcall–fsum.asm
In __fastcall, the biggest visual difference is the name decoration (name mangling): an underscore is prefixed, and then an @ sign, after which the length of the argument list, in bytes, is affixed.
The most important difference is that in __fastcall, the first integer argument is in ecx, the second integer argument in edx, and the rest will be on the stack. Since we only use one argument in the stack, the operand to the opcode ret is 4, not 12 like in the previous example.
Compilation and linking is straightforward:
And that concludes the second part of this series.
- Using win32 calling conventions, Unox Wiz, http://www.unixwiz.net/techtips/win32-callconv.html
- Calling conventions demystified, Code Project, https://www.codeproject.com/articles/1388/calling-conventions-demystified
- Calling conventions, MSDN, https://msdn.microsoft.com/en-us/library/k2b2ssfy.aspx