In BSidesSF CTF, calc.exe exploits you! (Author writeup of launchcode)

Hey everybody,

In addition to genius, whose writeup I already posted, my other favourite challenge I wrote for BSidesSF CTF was called launchcode. This will be my third and final writeup for BSidesSF CTF for 2019, but you can see all the challenges and solutions on our Github releases page.

This post will be more about how I developed this, since the solution is fairly straight forward once you know how it’s implemented. The idea came to me while my boyfriend, Chris, was playing a game on Steam Link. Due to an odd bug, while playing Dark Souls, the Steam Link kept shutting off. Eventually, we realized that even though the game was running, the Steam menu was still “in focus”. That means that while he was playing the game, he was also moving around and clicking things on the menu. Very reminiscent of Clickjacking attacks.

I tried to figure out something I could do with this, and slept on it. The next day, I had an idea: I’d write a backdoor into Windows Calculator such that, when a special code was entered, it’d decrypt and display the flag. Just like Steam Link, you’d be entering the code in the background as you’re doing calculations.

I really wanted some Windows reverse engineering challenges, not to mention oldschool software references, so this was perfect!

You can grab the patched binary and try it out, if you like! The code for that version is 9*4+0839, and the flag looks like this:

Getting calc.exe

I initially tried using c:\windows\system32\calc.exe from Windows 7, since that’s the version I had handy. I found the place where I wanted to add the custom code, then made a small change to make sure it’d work, and it started crashing immediately. Even changing a simple string prevented it from starting. I’m assuming it was due to code signing, but I’m not 100% sure. It also required the en/ or en-US/ folder to be present, which was annoying. I quickly gave up on that.

I decided to use Windows 2003 instead. I had the ISO for the OS handy, but didn’t really want to install it. I had no idea how Windows was packaged, so I just mounted the iso:

$ sudo mount -o loop,ro -t iso9660 Windows2003r2_Standard_SP2_x86_cd1.iso /mnt/tmp

Then explored the filesystem:

/mnt/tmp $ find . -name '*calc*'

Looks promising! What’s calc.ex_?

$ file ./i386/calc.ex_
./i386/calc.ex_: Microsoft Cabinet archive data, 41953 bytes, 1 file

This is just like a little CTF challenge!

Apparently I already had cabextract installed, so I tried it:

$ cabextract -d /tmp ./i386/calc.ex_
Extracting cabinet: ./i386/calc.ex_
  extracting /tmp/calc.exe

All done, no errors.
$ file /tmp/calc.exe
/tmp/calc.exe: PE32 executable (GUI) Intel 80386, for MS Windows

Huh, that was easy! You can download the base calc.exe if you want to follow along pre-modification.

Finding a place for code

I spent a lot of time coming up with ideas of how to backdoor calc.exe. I could have used a tool like Backdoor Factory, but that wasn’t quite right! I also had the idea of using debug hooks and causing exceptions to gain control, but I couldn’t get that to work.

Instead, I decided to find a place right in the binary where I could put a decent chunk of assembly code - like 150 bytes or so - that could handle all the logic. Then I’d just find the part of the code that handles “button pressed”, and call my backdoor code. That way, each button press, I could do some arbitrary calculation, then return control as if nothing had ever happened.

The question is, where? How do I spot useless code? In particular, in an executable segment?

TL;DR: I just wrecked up built-in security functionality. :)

I scrolled around for awhile, trying to think of where to put it. Then I noticed this:

.text:01007E2C ; =============== S U B R O U T I N E =======================================
.text:01007E2C ; Attributes: library function
.text:01007E2C ; __fastcall __security_check_cookie(x)
.text:01007E2C @__security_check_cookie@4 proc near    ; CODE XREF: sub_1001B19+470p
.text:01007E2C                                         ; sub_100209C+2E0p ...
.text:01007E2C var_2AC         = byte ptr -2ACh
.text:01007E2C                 cmp     ecx, ___security_cookie
.text:01007E32                 jnz     short $failure$18941
.text:01007E34                 retn

__security_check_cookie()? Who the heck needs a stack cookie for Windows Calculator, right? What are you going to do, hack yourself? :)

So I went ahead and changed the first byte of the function (0x01007E2C is the virtual address, or 0x722c in the file) to 0xc3 - ret. Then I ran calc.exe and verified that everything still worked. In theory, it’s slightly less secure - hah.

The rest of the function - 0x01007E2D up to 0x01007EC9 (156 bytes) - was now entirely unused. Perfect!

Finding a place for data

Next, I needed a part of memory that I can read/write to store the encrypted flag (so I could decrypt it later). There are plenty of easier ways to do this - such as by decrypting it on the stack - but this seemed easy enough to do. I also needed a smaller piece of unused memory to store the count (which is incremented each time the correct button is pressed - we’ll see that later).

In the start() function, I noticed this code:

.text:01012986                 push    offset unk_1014010
.text:0101298B                 push    offset unk_101400C
.text:01012990                 call    _initterm

I don’t care about terminal stuff, so maybe I could just kill that code and use that memory? While writing this blog, I actually looked up what initterm is, and realized that it isn’t even doing anything - it just points to two NULL addresses.

So I went ahead and NOP’ed out that code, so those two memory addresses would remain untouched. Then I set the value at that address to 0x0000000, to initialize the counter (although I think it already had that value).

For the actual flag, I had to store it in UTF-16, so if I wanted a 16 character flag, I needed 32 bytes of memory to store it in. That’s a lot!

I solved that problem by overwriting random unused-looking chunks of memory until calc.exe could start and avoid crashing. Probably not the cleanest solution - I may have redefined math as we know it - but it did the trick! That memory ended up starting here:

.data:010140C0                 db    0
.data:010140C1                 db    0
.data:010140C2                 db 0FFh
.data:010140C3                 db    0
.data:010140C4                 db  50h ; P
.data:010140C5                 db    0
.data:010140C6                 db    0
.data:010140C7                 db    0
.data:010140C8                 db 0FFh
.data:010140C9                 db    0
.data:010140CA                 db    0
.data:010140CB                 db    0
...32 bytes

AFAICT, that data is never read (but it’s certainly possible I’m wrong). I initialized that to the “encrypted” version of the flag, which is simply the 32 byte flag XORed by the values for certain calculator buttons - we’ll see that later.

So now we have space for code, a small amount of r/w memory (for a counter), and a larger amount of r/w memory (for the encrypted flag)! This could certainly have been done more easily, but I was finding memory as I needed it, so it became somewhat complex. Since this is a reverse engineering problem, that makes it all the more fun!

Find where button presses are handled

Now that we have space for code, how do we run the code?

I wish I had a good story about how to find the code. In the Windows 7 version of calc.exe, I used the compiled-in symbols to quickly narrow it down. But the Windows 7 version didn’t work, and Windows 2003 didn’t have symbols.

In the end, I literally just sorted the list of functions by length, and looked at the longest one: sub_10028A1.

The body of that sub looks like:

.text:010028C9                 jb      short loc_10028D3
.text:010028CB                 cmp     esi, 8Dh
.text:010028D1                 jbe     short loc_100292E
.text:010028D3 loc_10028D3:                            ; CODE XREF: sub_10028A1+28j
.text:010028D3                 cmp     esi, 132h
.text:010028D9                 jb      short loc_10028E3
.text:010028DB                 cmp     esi, 135h
.text:010028E1                 jbe     short loc_100292E
.text:010028E3 loc_10028E3:                            ; CODE XREF: sub_10028A1+38j
.text:010028E3                 cmp     esi, 136h
.text:010028E9                 jb      short loc_10028F3
.text:010028EB                 cmp     esi, 139h
.text:010028F1                 jbe     short loc_100292E
.text:010028F3 loc_10028F3:                            ; CODE XREF: sub_10028A1+48j
.text:010028F3                 cmp     esi, 13Ah
.text:010028F9                 jb      short loc_1002903
.text:010028FB                 cmp     esi, 13Ch
.text:01002901                 jbe     short loc_100292E

Compare, jump, compare, jump, compare jump, etc. That looks like buttons being checked! The value being compared at each step is stored in esi, which is defined at the top of that function:

.text:010028B7                 push    esi
.text:010028B8                 mov     esi, [ebp+hMem]
.text:010028BB                 mov     [ebp+var_14], eax
.text:010028BE                 mov     eax, 8Ch
.text:010028C3                 cmp     esi, eax

To figure out what that value is, I set a breakpoint at 0x010028BB:

0:000> bp 0x010028BB
0:000> g

Then I clicked on ‘0’, and execution stopped! I used r esi to see the argument:

Breakpoint 0 hit
010028bb 8945ec          mov     dword ptr [ebp-14h],eax ss:002b:000cfaa8=0100481e
0:000:x86> r esi
0:000:x86> g

Then I tried again with pressing ‘1’:

010028bb 8945ec          mov     dword ptr [ebp-14h],eax ss:002b:000cfaa8=0100481e
0:000:x86> r esi

I tried a few more just to make sure, but it worked as as you’d expect: 2 was 0x7e, 3 was 0x7f, 4 was 0x80, and so on. Wonderful!

Now that I know where button-pushes are processed, I know where I need to inject my code!

Hooking execution

Right at the start of sub_10028A1, the first two lines are:

.text:010028A1 B8 4A 2D 01 01                 mov     eax, offset sub_1012D4A
.text:010028A6 E8 F5 01 01 00                 call    sub_1012AA0
.text:010028AB 81 EC D0 00 00+                sub     esp, 0D0h

I’m not 100% sure what those first two lines are doing, and I still don’t really know, but I do know that if I NOP out that code, everything still works. Perfect! I assume it’s part of the “this execution is taking too long” code, since that stopped working when I added this patch. But we fix that later. :)

The space I freed up for my code starts at 0x01007E2D - one byte past the start of __security_check_cookie, where we created a fake ret instruction.

I want to change call sub_1012AA0 to call sub_1007E2D. The first byte of the original call - E8 - simply means “call a 4-byte relative address”. The next 4 bytes are the offset from the start of the next instruction to the instruction we want to call (as little endian).

What’s that mean? It means that we need to know how far it is from the next instruction (0x010028AB) and the place we want to call (0x01007E2D). We can figure that out with simple math:

0x01007E2D - 0x010028AB = 0x00005582

So we simply change the instruction to “E8 82 55 00 00” - “call sub_1007E2D”.

Of course, right now there’s just the remnants of the old function there, so if you call it, it’ll crash. That brings us to the last major part - the payload!


The payload is the most important part! It’s simply arbitrary assembly code that’ll run each time a button is pressed.

The way it works is, it takes the button code (such as 0x7c for ‘0’) and compares it to the current byte of the expected code (starting at offset 0). If it doesn’t match, we reset the counter and return. If it does match, we increment the counter and return if the counter is less than 8.

When the counter reaches 8 - the length of the code - it runs the “decryption” code and pops up a MessageBox with the decrypted flag. The decryption code will XOR the first 4 bytes of the encrypted flag with the first byte of the code. The next 4 bytes with the next byte, and so on, until the 8 code bytes have decoded the 32 flag bytes.

Note: this is not particularly good encryption, especially since the key is stored in memory. Don’t do that if you want real security! :)

Here’s the full, annotated assembly:

; eax = current character
mov eax, [ebp+8]

call get_values
  ; This is the expected code

  ; Get a handle to our "checker"
    pop esi

  ; Get the "index"
  mov ecx, [0x0101400c]

  ; Go to the "index"
  mov edx, esi
  add edx, ecx

  ; Check if we're good
  cmp byte [edx], al

  jz its_good

  ; If it's wrong, reset the count
  mov dword [0x0101400c], 0

  ; If it's right, increment the index
  inc ecx
  mov [0x0101400c], ecx

  ; Check if we're done
  cmp ecx, 0x08
  jl notdone

  ; If we ARE done, decode the actual flag
  ; esi = already the decoder string
  ; edi = the text
  mov edi, 0x010140c0
  xor ecx, ecx

; A little decoder loop
  mov al, [esi+ecx] ; al = this byte of the decoder string

  mov edx, ecx ; Multiply ecx by 4, so we can index into the encoded key
  shl edx, 2

  xor [edi+edx+0], al ; First byte
  xor [edi+edx+1], al ; Second byte
  xor [edi+edx+2], al ; Third byte
  xor [edi+edx+3], al ; First byte

  ; Increment the loop counter
  inc ecx
  cmp ecx, 0x08
  jl top

  ; Call MessageBoxW()!
  push 0
  push 0x010140c0 ; Decrypted flag
  push 0x010140c0 ; Decrypted flag
  push 0
  call [0x010011a8] ; MessageBoxW()


You may note two odd things: first, we call MessageBoxW(). MessageBoxW() is the UTF-16 messagebox function in Windows. That means we need to use 2 bytes for every character of our flag. Why do we use that one? Because it’s what calc.exe already imports - unfortunately, we can’t easily get MessageBoxA() without using GetProcAddress() or some other trick.

Second, the expected code is set to “XXXXXXXX”. In the patch.rb file, I actually generate the code entirely randomly, encrypt the flag with it, and dynamically fill in the flag and the code!

Here’s what it looks like when I run the generator script:

$ make
ruby ./do_patch.rb
Raw code: 5c 84 7e 5b 81 83 7f 84
Code: + 8 2 * 5 7 3 8
Code: ["5c847e5b81837f84"]
Encoded flag: 1f 5c 08 5c c2 84 ff 84 32 7e 3f 7e 0e 5b 15 5b c2 81 c9 81 c6 83 c7 83 5e 7f 01 7f f9 84 84 84
Patching 5 bytes @ 0x1ca6 from (inline data)
Patching 1 bytes @ 0x722c from (inline data)
Patching 15 bytes @ 0x11d86 from (inline data)
Patching 115 bytes @ 0x722d from realpatch.bin
Patching 4 bytes @ 0x1300c from (inline data)
Patching 32 bytes @ 0x130c0 from (inline data)
Patching 8 bytes @ 0x7236 from (inline data)
Patching 16 bytes @ 0x3be6 from (inline data)

It generated the code of +82*5738 - not quite so interesting, but that code is then written to 0x7236, which means it’s written directly over the “XXXXXXXX” string!

All said and done, the patch is 115 bytes long (with no real attempt to optimized my assembly). Not too shabby!

Fixing "this calculation is taking too long"

One funny thing - I was forever getting “This calculation is taking too long, would you like to continue?” messages. I tracked down the code to display it, and figured out it runs in a separate thread:

.text:010047E1 75 18                          jnz     short loc_10047FB
.text:010047E3 8D 45 FC                       lea     eax, [ebp+ThreadId]
.text:010047E6 50                             push    eax             ; lpThreadId
.text:010047E7 56                             push    esi             ; dwCreationFlags
.text:010047E8 56                             push    esi             ; lpParameter
.text:010047E9 68 1C 47 00 01                 push    offset StartAddress ; lpStartAddress
.text:010047EE 56                             push    esi             ; dwStackSize
.text:010047EF 56                             push    esi             ; lpThreadAttributes
.text:010047F0 FF 15 2C 10 00+                call    ds:CreateThread

By this point, I just wanted to go to bed (I wrote this whole thing in one evening!), so I literally just replaced that whole block of code - the pushes and the call - with NOPs. That thread no longer runs, and my problem is solved. :)

I still have no idea why I started seeing that.. presumably, I accidentally killed the reset-timer code.

Putting it all together

All said and done, here are all the patches I just talked about, all in one place:

patches = [
  # This patch calls the handler code at each character press by adding a "call"
  { offset: 0x1ca6,  max: 0x05, data: "\xe8\x82\x55\x00\x00" },

  # This patch gets rid of the "real" functionality by making it simply "return"
  { offset: 0x722c,  max: 0x01, data: "\xc3" },

  # This patch removs some console i/o stuff, giving us 20 bytes of freed-up
  # r/w memory where we store the counter
  { offset: 0x11d86, max: 0x0f, data: "\x90" * 0x0f },

  # This is the actual binary patch, which adds all the functionality (except
  # for the encoded flag)
  { offset: 0x722d,  max: 0x9c, file: 'realpatch.bin' },

  # This is the counter for the current byte we're at
  { offset: 0x1300c, max: 0x04, data: "\0\0\0\0" },

  # This is the r/w "encrypted" data (in maybe-unused memory)
  { offset: 0x130c0, max: 0x20, data: ENCODED_FLAG },

  # This is the r/o "validator" data (it's 9 past the start of the realpatch data)
  { offset: 0x722d+9, max: 0x08, data: CODE },

  # This kills the annoying "This operation is taking too long!!!" thread
  { offset: 0x3be6, max: 0x10, data: "\x90" * 0x10 },

We replace the call to get control of code execution; we disable stack cookies to free up code space; we disable console i/o stuff to free up data stuff; we overwrite the stack cookie function with realpatch.bin; we reset the counter; we copy our encrypted flag to probably-unused memory; we patch the code directly into the assembly; and finally, we kill the “operation is taking too long” thread.

Eight patches, and we have a cool backdoor in Calc! The coolest thing is, the flag and code are both generated dynamically. That means you can easily change the code and data and get your own encrypted flag!

Solving it

I honestly haven’t solved it myself, other than typing in the code to verify it works, but I talked to one team and asked how they solved it.

Their answer: “we diffed it with the real version, and the code stuck out like a sore thumb!”

Once you know where the hook is, the assembly code I posted above is pretty easy to reverse engineer for somebody experienced in doing CTFs. Just gotta debug it to figure out how the codes (like 0x7C) correspond to buttons (‘0’)!


Join the conversation on this Mastodon post (replies will appear below)!

    Loading comments...