Before telling you what this post is, let me tell you what this is not:
- It is not a step-by-step guide to Assembly language programming.
- It is not a step-by-step guide to Win32 programming.
- It is not a replacement for the official NASM documentation.
- It is not a replacement for the official GCC documentation.
Having said that, let me tell you what this post is: this post is a collection of my experiences, the sum total of my mistakes, hard earned, while trying to program in assembly, C and Win32. Hopefully, someone will find it useful.
In order to understand and gain the maximum use out of this series, you will need:
- Good experience in C programming
- Some experience in calling Win32 API (from within C/C++ or other language)
- Some exposure to assembly programming, hopefully in NASM
Assembly programming is hard. Win32 programming is hard. Mixing them both is harder. Add to that the 32-bit/64-bit mix-ups and it’s going to be a veritable nightmare. That is why this blog exists. It will outline where I’ve gone wrong, and why I’ve gone wrong. But most importantly, it will log how I’ve corrected those mistakes.
On we go, then.
Getting the Tools
I like to use Notepad++ as my primary editor, but when I’m doing multi-file Windows programming it makes much sense to use DevCPP. Keep in mind that DevCPP uses MinGW port of GCC under the hood. You might need that fact later.
There are a good lot of assemblers out there. I was strongly tempted to try Microsoft Macro Assembler (MASM), but in the end I decided against it. After all, if I wanted it easy, I’d have gone for C#.NET: the whole purpose of this excursion was to take a deep dive. Reminding myself of that, I chose to go with Netwide Assembler. In all programs in this blog, I will be using version 2.11.05, but you’re free to download the latest version here.
Although I am not a big fan of the argument “Command line tools build character”, I decided not to go with IDE tools for assembly. However, there are good tools out there, and I can mention at least one here: SASM. You can download for Windows here.
I decided to go with gcc, simply for the sake of trust on the community. I use version 5.1.0 in this document.
Since I chose the GCC suite, I already get ld for free. But if you want an alternative, I can suggest ALINK.
“Hello, World”–from NASM, the Wrong Way
I’m sure most of you have seen some version of the following code as the “first Hello, World” program.
This will work fine in Linux and pretty much any *nix. But, the sad news is that this won’t work under Windows at all! Why? Because you’re calling the Linux interrupt code (syscall) 0x80, which doesn’t exist in Windows. In Windows (or rather, DOS), the correct interrupt function would be 0x21 (or 21h), but that would mean you’re forced to write in 16-bit. Either way, calling the kernel directly in Windows seems to be not the way forward.
If we cannot directly call the kernel, then what options are available to us? The obvious choice is C runtime libraries. And why not the trusted printf? Let’s see that version.
“Hello, Windows”–Take Two, with printf
This time, we call printf. We’ll have a deep look at the way the arguments are passed in Part 2 of this article.
You need to compile this with the following line.
And then link it with gcc like so:
Remember that both NASM and GCC succeed silently. That means, unless there’s an error, you get no output on console.
Couple of points to note here:
- It is important to use the flag -f win32 here. (Both -fwin32 and -f win32 will work.) Unless you do so, NASM will happily try to compile your assembly file into a binary format (*.bin), find that it has an external reference, and fail with “error: binary output does not support external references“.
- It is important to use the option -m32 with gcc here. (Unlike nasm, gcc will not let you put a space between -m and 32.) If you do not specify -m32, then gcc will try to build a 64-bit exe, fail, and complain that “i386 architecture of input file hello.obj is incompatible with i386:x86-64 output.” In addition, you’ll get an error saying “undefined reference to WinMain“.
- The -o option lets gcc know the name of the output file. Quite inconsistently, here, gcc will not mind the space between -o and hello.exe. If you don’t specify the output file name, you’ll get a file called “a.exe”.
Assuming everything went right, you should get a file called “hello.exe” in your directory, which you can execute like so:
Cheers! You just called a C library routine from assembly, and made some basic I/O work happen.
However, note that we’re still using DOS subsystem. Our aim was to program for Win32, not call a C routine. Let’s do so now.
“Hello, World”–Take Three, with _WriteFile@20
This time, we’re going to use Win32 API to directly access the console.
Like in the previous examples, you have to assemble, link and run. However, let me introduce another way to do all 3 in one line:
If all went well, you should see something like this:
Again, couple of important points:
- Where is GetStdHandle declared in? The answer is, Kernel32.dll. The immediate next question is, how did gcc know to link with Kernel32.lib? Aren’t we supposed to get an error like this? The short answer is, because we specified the -m32 flag.
- Why the funny names, like _GetStdHandle@4? The answer has to do something with the way method names are mangled or decorated in Win32. That’s juicy material for a next article. For now, remember that the calling convention for Win32 is known as __stdcall, which defines the way functions are made available to public after compilation. Specifically, the MSDN article says, that under __stdcall, “an underscore (_) is prefixed to the name. The name is followed by the at sign (@) followed by the number of bytes (in decimal) in the argument list.“
- Why the constant (-11)? That’s how we tell _GetStdHandle@4 to get us the “Standard Output” (which in this case, is the screen). Consider this the equivalent way of grabbing a handle to stdout in C, and cout in C++.
Again, knowing how a Win32 function is mangled, or decorated after compiling seems like far too much to expect at this level. After all, all C programmers get to happily write “ExitProcess” instead of “_ExitProcess@4”.
In the next step, we will look at how this can be done.
“Hello, World”–Take Four, with WriteFile
I would like to warn you beforehand: we’ll run into a (rather annoying) known issue with NASM here. This will force us to so something out of the ordinary. Also, this will force us to use ALINK instead of gcc/ld as our linker.
Here’s the code:
Couple of things to note:
- Note that this is identical to the previous version, Take #3, except for this section:
What this does is to import function names directly as defined in the DLLs, thereby freeing us from having to mangle the function names ourselves.
- Note that we’re using the function names within square brackets, like to: [ExitProcess]
However, if you try to assemble this with NASM in the usual way, using -fwin32, NASM will throw an error:
Unfortunately, there is nothing we can do about this, so we will use a workaround. We are going to use -fobj instead of -fwin32. At least this will give us an obj file.
Now, if we use our usual way of gcc to link this, we will get an error.
This means that ld (which is the linker under the hood of gcc) did not like our obj format. And it’s right. We did indeed supply a wrong file format. What we should now do is to find a less restricting linker that will overlook this fact. Enter alink.
Here, -subsys can have two values: console and gui. Since this is a console application, we will go with -subsys console. Also, similar to -m32 in gcc, we will need to specify that we want a Win32 PE format executable file: hence the flag -oPE.
If all went well, you should get an output like so:
Now, we will finally look at how to go to the other mode, the GUI mode. The simplest way of displaying text output in Win32 is MessageBox function, which comes in 2 flavors: MessageBoxA for ANSI strings and MessageBoxW for UNICODE strings. We will go with ANSI version for now.
“Hello, World”–Take Five, with MessageBoxA
We assemble, link and execute in one step, like so:
Note the flag -subsys -gui to alink. There will be no console output, but you should see a familiar message box popping up.
This concludes the first part. In the next part, we’ll take a deeper look at interfacing with C library functions, with special attention to what are known as calling conventions.