Win32 Programming in NASM – Part I

Introduction

Before telling you what this post is, let me tell you what this is not:

  • It is not a step-by-step guide to Assembly language programming.
  • It is not a step-by-step guide to Win32 programming.
  • It is not a replacement for the official NASM documentation.
  • It is not a replacement for the official GCC documentation.

Having said that, let me tell you what this post is: this post is a collection of my experiences, the sum total of my mistakes, hard earned, while trying to program in assembly, C and Win32. Hopefully, someone will find it useful.

In order to understand and gain the maximum use out of this series, you will need:

  1. Good experience in C programming
  2. Some experience in calling Win32 API (from within C/C++ or other language)
  3. Some exposure to assembly programming, hopefully in NASM

Assembly programming is hard. Win32 programming is hard. Mixing them both is harder. Add to that the 32-bit/64-bit mix-ups and it’s going to be a veritable nightmare. That is why this blog exists. It will outline where I’ve gone wrong, and why I’ve gone wrong. But most importantly, it will log how I’ve corrected those mistakes.

On we go, then.

Getting the Tools

Editors

I like to use Notepad++ as my primary editor, but when I’m doing multi-file Windows programming it makes much sense to use DevCPP. Keep in mind that DevCPP uses MinGW port of GCC under the hood. You might need that fact later.

Assembler

There are a good lot of assemblers out there. I was strongly tempted to try Microsoft Macro Assembler (MASM), but in the end I decided against it. After all, if I wanted it easy, I’d have gone for C#.NET: the whole purpose of this excursion was to take a deep dive. Reminding myself of that, I chose to go with Netwide Assembler. In all programs in this blog, I will be using version 2.11.05, but you’re free to download the latest version here.

Although I am not a big fan of the argument “Command line tools build character”, I decided not to go with IDE tools for assembly. However, there are good tools out there, and I can mention at least one here: SASM. You can download for Windows here.

C Compiler

GCC or MinGW? You can read all about the difference between GNU, GCC and MinGW in this excellent stackoverflow question (and this as well).

I decided to go with gcc, simply for the sake of trust on the community. I use version 5.1.0 in this document.

Linker

Since I chose the GCC suite, I already get ld for free. But if you want an alternative, I can suggest ALINK.

“Hello, World”–from NASM, the Wrong Way

I’m sure most of you have seen some version of the following code as the “first Hello, World” program.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
section .text
global _start
_start: 
mov ecx,msg 
mov ebx,1 
mov eax,4 
int 0x80 

mov eax,1 
int 0x80 

section .data

msg db 'Hello, Windows!',0xa 
len equ $ - msg 

This will work fine in Linux and pretty much any *nix. But, the sad news is that this won’t work under Windows at all! Why? Because you’re calling the Linux interrupt code (syscall) 0x80, which doesn’t exist in Windows. In Windows (or rather, DOS), the correct interrupt function would be 0x21 (or 21h), but that would mean you’re forced to write in 16-bit. Either way, calling the kernel directly in Windows seems to be not the way forward.

If we cannot directly call the kernel, then what options are available to us? The obvious choice is C runtime libraries. And why not the trusted printf? Let’s see that version.

“Hello, Windows”–Take Two, with printf

This time, we call printf. We’ll have a deep look at the way the arguments are passed in Part 2 of this article.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    global  _main
    extern  _printf

    section .text
_main:
    push    message
    call    _printf
    add     esp, 4
    ret
message:
    db  'Hello, World', 10, 0

You need to compile this with the following line.

C:\work>nasm -f win32 hello.asm

And then link it with gcc like so:

C:\work>gcc -m32 hello.obj -o hello.exe

Remember that both NASM and GCC succeed silently. That means, unless there’s an error, you get no output on console.

Couple of points to note here:

  1. It is important to use the flag -f win32 here. (Both -fwin32 and -f win32 will work.) Unless you do so, NASM will happily try to compile your assembly file into a binary format (*.bin), find that it has an external reference, and fail with “error: binary output does not support external references“.
  2. It is important to use the option -m32 with gcc here. (Unlike nasm, gcc will not let you put a space between -m and 32.) If you do not specify -m32, then gcc will try to build a 64-bit exe, fail, and complain that “i386 architecture of input file hello.obj is incompatible with i386:x86-64 output.” In addition, you’ll get an error saying “undefined reference to WinMain“.
  3. The -o option lets gcc know the name of the output file. Quite inconsistently, here, gcc will not mind the space between -o and hello.exe. If you don’t specify the output file name, you’ll get a file called “a.exe”.

Assuming everything went right, you should get a file called “hello.exe” in your directory, which you can execute like so:

C:\work>hello
Hello, World
C:\work>

Cheers! You just called a C library routine from assembly, and made some basic I/O work happen.

However, note that we’re still using DOS subsystem. Our aim was to program for Win32, not call a C routine. Let’s do so now.

“Hello, World”–Take Three, with _WriteFile@20

This time, we’re going to use Win32 API to directly access the console.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
    global _main
    extern  _GetStdHandle@4
    extern  _WriteFile@20
    extern  _ExitProcess@4

    section .text
_main:
    ; DWORD  bytes;    
    mov     ebp, esp
    sub     esp, 4

    ; hStdOut = GetstdHandle( STD_OUTPUT_HANDLE)
    push    -11
    call    _GetStdHandle@4
    mov     ebx, eax    

    ; WriteFile( hstdOut, message, length(message), &bytes, 0);
    push    0
    lea     eax, [ebp-4]
    push    eax
    push    (message_end - message)
    push    message
    push    ebx
    call    _WriteFile@20

    ; ExitProcess(0)
    push    0
    call    _ExitProcess@4

    ; never here
    hlt
message:
    db      'Hello, World', 10
message_end:

Like in the previous examples, you have to assemble, link and run. However, let me introduce another way to do all 3 in one line:

1
C:\work>nasm -fwin32 hellow.asm && gcc -m32 hellow.obj -o hellow.exe && hellow

If all went well, you should see something like this:

1
2
3
C:\work>nasm -fwin32 hellow.asm && gcc -m32 hellow.obj -o hellow.exe && hellow
Hello, World
C:\work>

Again, couple of important points:

  1. Where is GetStdHandle declared in? The answer is, Kernel32.dll. The immediate next question is, how did gcc know to link with Kernel32.lib? Aren’t we supposed to get an error like this? The short answer is,  because we specified the -m32 flag.
  2. Why the funny names, like _GetStdHandle@4? The answer has to do something with the way method names are mangled or decorated in Win32. That’s juicy material for a next article. For now, remember that the calling convention for Win32 is known as __stdcall, which defines the way functions are made available to public after compilation. Specifically, the MSDN article says, that under __stdcall, “an underscore (_) is prefixed to the name. The name is followed by the at sign (@) followed by the number of bytes (in decimal) in the argument list.
  3. Why the constant (-11)? That’s how we tell _GetStdHandle@4 to get us the “Standard Output” (which in this case, is the screen). Consider this the equivalent way of grabbing a handle to stdout in C, and cout in C++.

Again, knowing how a Win32 function is mangled, or decorated after compiling seems like far too much to expect at this level. After all, all C programmers get to happily write “ExitProcess” instead of “_ExitProcess@4”.

In the next step, we will look at how this can be done.

“Hello, World”–Take Four, with WriteFile

I would like to warn you beforehand: we’ll run into a (rather annoying) known issue with NASM here. This will force us to so something out of the ordinary. Also, this will force us to use ALINK instead of gcc/ld as our linker.

Here’s the code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[BITS 32]

extern ExitProcess
import ExitProcess kernel32.dll

extern  GetStdHandle
import GetStdHandle kernel32.dll

extern  WriteFile
import WriteFile kernel32.dll

segment .data use32
Text db "Hello, World",0

segment .code use32
..start:
    push    -11
    call    	[GetStdHandle]
    mov     ebx, eax  
	
    ; WriteFile( hstdOut, message, length(message), &bytes, 0);
    push    dword 0
    lea     	eax, [ebp-4]
    push    eax
    push    (message_end - message)
    push    message
    push    ebx
    call    	[WriteFile]

    ; ExitProcess(0)
    push    dword 0
    call    	[ExitProcess]

    ; never here
    hlt
message:
    db      'Hello, World', 10
message_end:

Couple of things to note:

  1. Note that this is identical to the previous version, Take #3, except for this section:

    1
    2
    3
    4
    5
    6
    7
    8
    extern ExitProcess
    import ExitProcess kernel32.dll
    
    extern  GetStdHandle
    import GetStdHandle kernel32.dll
    
    extern  WriteFile
    import WriteFile kernel32.dll
    

    What this does is to import function names directly as defined in the DLLs, thereby freeing us from having to mangle the function names ourselves.

  2. Note that we’re using the function names within square brackets, like to: [ExitProcess]

However, if you try to assemble this with NASM in the usual way, using -fwin32, NASM will throw an error:

C:\work>nasm -fwin32 hellow2.asm
hellow2.asm:4: error: parser: instruction expected
hellow2.asm:7: error: symbol `import' redefined
hellow2.asm:7: error: parser: instruction expected
hellow2.asm:10: error: symbol `import' redefined
hellow2.asm:10: error: parser: instruction expected

Unfortunately, there is nothing we can do about this, so we will use a workaround. We are going to use -fobj instead of -fwin32. At least this will give us an obj file.

C:\work>nasm -fobj hellow2.asm

Now, if we use our usual way of gcc to link this, we will get an error.

C:\work>gcc -m32 hellow2.obj -o hellow2.exe
hellow2.obj: file not recognized: File format not recognized
collect2.exe: error: ld returned 1 exit status

This means that ld (which is the linker under the hood of gcc) did not like our obj format. And it’s right. We did indeed supply a wrong file format. What we should now do is to find a less restricting linker that will overlook this fact. Enter alink.

C:\work>alink -subsys console -oPE hellow2.obj
ALINK v1.6 (C) Copyright 1998-9 Anthony A.J. Williams.
All Rights Reserved

Loading file hellow2.obj
matched Externs
matched ComDefs
Generating PE file hellow2.exe

C:\work>

Here, -subsys can have two values: console and gui. Since this is a console application, we will go with -subsys console. Also, similar to -m32 in gcc, we will need to specify that we want a Win32 PE format executable file: hence the flag -oPE.

If all went well, you should get an output like so:

C:\work>hellow2
Hello, World
C:\work>

Good!

Now, we will finally look at how to go to the other mode, the GUI mode. The simplest way of displaying text output in Win32 is MessageBox function, which comes in 2 flavors: MessageBoxA for ANSI strings and MessageBoxW for UNICODE strings. We will go with ANSI version for now.

“Hello, World”–Take Five, with MessageBoxA

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
[BITS 32]

extern ExitProcess
import ExitProcess kernel32.dll
extern MessageBoxA
import MessageBoxA user32.dll

segment .data use32
Caption db 'From Assembly',0
Text db "Hello, World",0

segment .code use32
..start:
push dword 0
push dword Caption
push dword Text
push dword 0
call [MessageBoxA]

push dword 0
call [ExitProcess]

We assemble, link and execute in one step, like so:

I:\Work\Shellcode\Samples\0>nasm -fobj hellowin.asm && alink -subsys gui -oPE he
llowin.obj && hellowin
ALINK v1.6 (C) Copyright 1998-9 Anthony A.J. Williams.
All Rights Reserved

Loading file hellowin.obj
matched Externs
matched ComDefs
Generating PE file hellowin.exe

I:\Work\Shellcode\Samples\0>

Note the flag -subsys -gui to alink. There will be no console output, but you should see a familiar message box popping up.

This concludes the first part. In the next part, we’ll take a deeper look at interfacing with C library functions, with special attention to what are known as calling conventions.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s