Win32 Programming in NASM – Part II

Introduction

In Part 1 of this series, we proceeded from a simple, console-based “Hello, World” to a gui-based version where we directly call Win32 functions from NASM. In that section, we postponed discussing a few important things, like calling convention. We will look at that in depth in this second part.

C calling ASM, Please Respond

Let’s jump right in and do a very small task here: we’ll write a function that adds two integers in assembly, and call that function in C. I’m going to create 2 files for this exercise, asum.asm and sum.c.

Here is my assembly code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
; asum.asm
[BITS 32]

global _asum

section .text

_asum:
    push   ebp                   ; preserve ebp
    mov        ebp, esp            ; save stack pointer to be used as a base pointer
    mov        eax, [ebp + 8]   ; first 4 bytes
    add        eax, [ebp + 12] ; add next 4 bytes
    pop        ebp                  ; pop the topmost value in stack to ebp
    ret

Things to note:

  1. We preserve ebp by pushing it into the stack. This is because we want to use ebp as our stack reference. After pushing ebp onto the stack, we then move esp to ebp, creating a stack frame.
  2. There is a way that C calls functions, known as C calling convention. In this convention, the caller pushes arguments onto the stack in a right-to-left manner. Therefore, the first argument to be passed to the function is the topmost on stack. This is useful in creating variable-argument list functions, like printf. As soon as the callee reads the first argument, it will have to know how many more arguments to expect. In this case, therefore, the C caller pushed the integers b and a (in that order) to the stack. On top of that, we pushed the old value of ebp onto the stack as well. So now the stack would look like this:

    |ebp           | <--- top of stack (esp)
    |return address|
    |a             |
    |b             |
    

    You can get more info about C calling convention (known as cdecl) here. Also, read the relevant NASM documentation here.

  3. Remember that the stack grows from high memory to low memory. Therefore, [ebp] now points to where we saved the old value of ebp (which is 8 bytes), [ebp+4] is the return address, pushed implicitly by the call instruction that called us, [ebp+8] is a 4-byte integer which is the first argument a, and [ebp+12] which too is a 4-byte integer, the second argument b.
  4. What we now do is pretty straightforward: we move the first argument to the accumulator, eax, add the second argument to eax. Now, eax has the sum of the two integers. We will leave it there because that’s where the calling C function expects to find it.
  5. Finally, we pop the old value of ebp from the top of stack. Since the number of pushes equal the number of pops, there are no stack adjustments to be made.
  6. Note that we didn’t clear the stack. This is again as per C calling convention, where clearing the stack is done by the caller, not callee.

The C code calling this function is pretty straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
#include <stdio.h>

extern int asum(int a, int b);

int main()
{
    int i;
    i = asum(1,2);
    printf("sum of 1 and 2 are: %i", i);
}

Notice that even though the assembly function is named “_asum“, in C, we define it to be “asum“. Why so? Because the C compiler will mangle, or decorate the function name with an underscore prefixed.

We then proceed to assemble, link and run, like so:

1
2
3
4
C:\work>nasm -felf asum.asm
C:\work>gcc -m32 asum.o sum.c -o sum.exe
C:\work>sum
The sum of 1 and 2 are: 3

Note that we chose elf as the format of the compiled assembly. This is because we want to keep the function name to “_asum” which then, can be called from C as simply “asum“.

ASM Calling C, Please Respond

Now that we had a look at how a C-callable assembly function could be written, the next logical step is to check if we can do the same vice versa: to call a C routine from within assembly.

In fact, if you look at Part 1 of this series, you might notice that the call to printf was exactly like that. Nevertheless, we will look into another example here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
[BITS 32]

    global  _main
    extern  _puts

    section .text
_main:
    push    message
    call    _puts
    add     esp, 4
    ret
message:
    db  'Ayubowan', 10, 0

In case you’re wondering, “Ayubowan” is the traditional welcome in Sinhalese language (which, if you’re curious, has a literal meaning of ‘may you live longer’).

I want you to note this statement: add esp, 4 . Why does the caller have to raise the stack pointer by 4 bytes?

Remember that puts accepts a pointer to a char array. That pointer in 32 bits is 4 bytes long. We, the caller, pushed that 4-byte integer value to the stack, to be picked up by puts. Now, once puts is done processing, we want to clear the stack by raising the esp (thus effectively clearing the stack).

Assembling and running is straightforward:

C:\work>nasm -fwin32 ayubowan.asm

C:\work>gcc -m32 ayubowan.obj -o ayubowan.exe

C:\work>ayubowan
Ayubowan


I:\Work\Shellcode\Samples\1>

Remember that this time, the driver program (or the program that has the main entry point) is assembly; this is why we had to run nasm with -fwin32. (In the previous example, we ran nasm with -felf and gcc with -m32.)

Calling Conventions

Calling conventions decide three important things:

  1. What happens to the stack, before and after a function call?
  2. What happens to the registers, before and after a function call?
  3. How is the function name decorated?

There are lots of calling conventions: __stdcall (Win32 default)and __cdecl (C/C++ default), in addition to a whole lot of others like __fastcall, __clrcall, __thiscall and __vectorcall (plus some obsolete ones like __pascal, __fortran and __syscall).

Now, before we go down to the all the implementations, let us quickly list down a handy shortcut table of the two most important conventions so that we all remember who does what.

Calling Convention __cdecl __stdcall
Argument passing left-to-right left-to-right
Argument passing convention By value, unless a pointer / reference type is passed By value, unless a pointer / reference type is passed
Who cleans up the stack caller callee
Name decoration Prefix underscore Prefix underscore, then @ sign, then the number of bytes to pop from stack

Now, the next task is for us to write an assembly function that gets called successfully from C, using any of these conventions, that adds 3 numbers and return the sum (to the caller). We will first look at __cdecl, and then __stdcall. After that, it is a minor change from __stdcall to __fastcall. In each case, a C main() method will call our assembly function using one of the 3 conventions.

When writing the C function, we come across one of those annoying differences between compilers. Imagine the __fastcall version: in Microsoft C comipler, the function prototype has to be something like this.

1
int __fastcall fsum(int a, int b, int c);

Whereas in GCC, it will have to be something like this:

1
int fsum(int a, int b, int c)  __attribute__((fastcall));

The difference has to do with how each compiler implements the calling conventions.

Luckily, we can use the preprocessor to ease our burden here a lot. Although not very pretty, this simple macro will certainly save the trouble.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
#if defined(_MSC_VER)
    #define MSCDECL __cdecl
    #define MSSTDCALL __stdcall
    #define MSFASTCALL __fastcall
    #define GCCCDECL
    #define GCCSTDCALL
    #define GCCFASTCALL
#elif defined(__GNUC__)
    #define MSCDECL
    #define MSSTDCALL
    #define MSFASTCALL
    #define GCCCDECL __attribute__((cdecl))
    #define GCCSTDCALL __attribute__((stdcall))
    #define GCCFASTCALL __attribute__((fastcall))
#endif

int MSFASTCALL fsum(int a, int b, int c) GCCFASTCALL;

I owe the answer to this stackoverflow question.
Now that we have a way to correctly mark our function prototype with the calling convention, writing the calling function is trivial:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// caller.c
#if defined(_MSC_VER)
    #define MSCDECL __cdecl
    #define MSSTDCALL __stdcall
    #define MSFASTCALL __fastcall
    #define GCCCDECL
    #define GCCSTDCALL
    #define GCCFASTCALL
#elif defined(__GNUC__)
    #define MSCDECL
    #define MSSTDCALL
    #define MSFASTCALL
    #define GCCCDECL __attribute__((cdecl))
    #define GCCSTDCALL __attribute__((stdcall))
    #define GCCFASTCALL __attribute__((fastcall))
#endif

int MSCDECL csum(int a, int b, int c) GCCCDECL;
int MSSTDCALL ssum(int a, int b, int c) GCCSTDCALL;
int MSFASTCALL fsum(int a, int b, int c) GCCFASTCALL;

void main()
{
    int a = 2, b = 3, c = 5;
    int cs = csum(a, b, c);
    int ss = ssum(a, b, c);
    int fs = fsum(a, b, c); 
    
    printf("csum says: the sum of %i, %i and %i are %i", a, b, c, cs);
    printf("ssum says: the sum of %i, %i and %i are %i", a, b, c, ss);
    printf("fsum says: the sum of %i, %i and %i are %i", a, b, c, fs);
}

The real task is now, writing the 3 routines csum, ssum and fsum in NASM, that honor these 3 calling conventions.

Sum via __cdecl–csum.asm

First up, is the easiest one: the cdecl version.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
; csum.asm
[BITS 32]

global _csum

section .text

_csum:
    push  ebp ; preserve ebp
    mov      ebp,esp
    xor   eax,eax
    add   eax,[ebp+8]
    add   eax,[ebp+12]
    add   eax,[ebp+16]
    pop   ebp ; pop the topmost value in stack to ebp
    ret

As seen here, the callee does not pop any parameters from stack, except the ebp it pushed for its own usage. The arguments from right-to-left are found at [ebp+12], [ebp+8] and [ebp+4], respectively. (Remember that ebp itself contains the return address.) The return value is left in eax, where the C routine finds it.

Sum via __stdcall–ssum.asm

In __stdcall, all arguments are pushed to the stack just the same way as in __cdecl, but this time, it is the callee’s responsibility to clean the stack. This is achieved by the operand to the ret opcode, which tells the number of bytes to pop from the stack.

Plus, the name decoration is: prefix underscore, function name, @ sign, and the the length of the argument list, in bytes. If the function takes no arguments (i.e. void), this should be 0.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
; ssum.asm
[BITS 32]

global _ssum@12

section .text

_ssum@12:
    push   ebp ; preserve ebp
    mov    ebp,esp
    xor    eax,eax
    add    eax,[ebp+8]
    add    eax,[ebp+12]
    add    eax,[ebp+16]
    pop    ebp ; pop the topmost value in stack to ebp
    ret    12

Also, note that the return value is left in eax, just like in __cdecl.

Sum via __fastcall–fsum.asm

In __fastcall, the biggest visual difference is the name decoration (name mangling): an underscore is prefixed, and then an @ sign, after which the length of the argument list, in bytes, is affixed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
; fsum.asm
[BITS 32]

global @fsum@12

section .text

@fsum@12:
    push   ebp ; preserve ebp
    mov    ebp,esp
    xor    eax,eax
    add    eax,ecx
    add    eax,edx
    add    eax,[ebp+8]
    pop    ebp ; pop the topmost value in stack to ebp
    ret    4

The most important difference is that in __fastcall, the first integer argument is in ecx, the second integer argument in edx, and the rest will be on the stack. Since we only use one argument in the stack, the operand to the opcode ret is 4, not 12 like in the previous example.

Compilation and linking is straightforward:

C:\work>nasm -fwin32 csum.asm
C:\work>
C:\work>nasm -fwin32 ssum.asm
C:\work>
C:\work>nasm -fwin32 fsum.asm
C:\work>
C:\work>gcc -m32 csum.obj ssum.obj fsum.obj caller.c -o caller.exe
C:\work>
C:\>work>caller
csum says: the sum of 2, 3 and 5 are 10
ssum says: the sum of 2, 3 and 5 are 10
fsum says: the sum of 2, 3 and 5 are 10

C:\>work>

And that concludes the second part of this series.

Additional Reading

  1. Using win32 calling conventions, Unox Wiz, http://www.unixwiz.net/techtips/win32-callconv.html
  2. Calling conventions demystified, Code Project, https://www.codeproject.com/articles/1388/calling-conventions-demystified
  3. Calling conventions, MSDN, https://msdn.microsoft.com/en-us/library/k2b2ssfy.aspx
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s