QBDI vs TritonDSE against a VM: who will be the fastest?

In this blog, we present how QBDI and TritonDSE can be used to attack a complex C++ binary implementing a VM.

In this blog, we present how QBDI and TritonDSE can be used to attack a complex C++ binary implementing a VM. Introduction The 3rd edition of the Jeanne d'Hack CTF took place on January 30–31, 2026, and I had the opportunity to beta test the reverse challenges. Challenges are based on a main binary, providing a kind of API to interact with the user (print content on screen, ask for question depending various choices, etc...), in an interactive console-mode game. Each level comes as a separate library. This blogpost will present two solutions, one with TritonDSE and the other with QBDI, to solve the last level ( level_4.so ), which implement a custom VM. +==============================================================================+ | The Dragon's Lair | +==============================================================================+ | |===-~___ _,-' | | -==\\ `//~\\ ~~~~` ---.___.-~' | | ______-==| | | \\ _-~` | | __--~~~ ,-/-==\\ | | `\ ,' | | _-~ /' | \\ / / \ / | | .' / | \\ /' / \ /' | | / ____ / | \` \.__/-~~ ~ \ _ _/' / \/' | |/-'~ ~~~~~---__ | ~-/~ ( ) /' _--~` | | \_ | / _) ; ), __--~~ | | '~~--_/ _-~/- / \ '-~ \ | | {\__--_/} / \\_>- )<__\ \ | | /' (_/ _-~ | |__>--<__| | | | |0 0 _/) )-~ | |__>--<__| | | | / /~ ,_/ / /__>---<__/ | | | o o _// /-~_ >---<__-~ / | | (^(~ /~_>---<__- _-~ | | /__>--<__/ _-~ | | |__>--<__| / .---_ | | |__>--<__| | /' _--_ ~\ | | |__>--<__| | /' / ~\`\| | \__>--<__\ \ /' // ||| | ~-__>--<_~-_ ~--____---~' _/'/ /' | | ~-_ ~>--<_/-__ __-~ _/ | | ~~-'_ /_/ /~~~~~~~__--~ | | ~~~~~~~~~~ | | | +==============================================================================+ | You venture deeper into the cave system. The air grows warmer | | with each step, and a faint red glow appears ahead. | | | | As you round a corner, the tunnel opens into a massive cavern. | | Your heart stops. In the center lies an enormous dragon, | | its golden scales reflecting the dim light. | | | +==============================================================================+ | [0] Try to sneak past quietly. | | [1] Attack while it's sleeping. | | [2] Back away slowly. | +------------------------------------------------------------------------------+ Analysis After playing the fourth level a few times, it appears that we always die when facing the dragon. There aren't many options, and all our actions seem to lead to the same result. Let's open this library in a disassembler. Looking at the xref to windows_frame brings us to one strange thing: .text: 000000000000 C93B call _create_choices .text: 000000000000 C940 mov [ rbp + var_18 ], rax .text: 000000000000 C944 mov rax , [ rbp + var_18 ] .text: 000000000000 C948 lea rdx , aStrikeAtItsHea ; "Strike at its heart." .text: 000000000000 C94F mov rsi , rdx .text: 000000000000 C952 mov rdi , rax .text: 000000000000 C955 call _choices_add .text: 000000000000 C95A mov rax , [ rbp + var_18 ] .text: 000000000000 C95E lea rdx , aAimForItsEyes ; "Aim for its eyes." .text: 000000000000 C965 mov rsi , rdx .text: 000000000000 C968 mov rdi , rax .text: 000000000000 C96B call _choices_add .text: 000000000000 C970 mov rax , [ rbp + var_18 ] .text: 000000000000 C974 lea rdx , aLookForAWeakne ; "Look for a weakness." .text: 000000000000 C97B mov rsi , rdx .text: 000000000000 C97E mov rdi , rax .text: 000000000000 C981 call _choices_add .text: 000000000000 C986 mov rax , [ rbp + var_48 ] .text: 000000000000 C98A mov rdi , rax .text: 000000000000 C98D call sub_C658 .text: 000000000000 C992 test eax , eax .text: 000000000000 C994 setz al .text: 000000000000 C997 test al , al .text: 000000000000 C999 jz short loc_C9B1 .text: 000000000000 C99B mov rax , [ rbp + var_18 ] .text: 000000000000 C99F lea rdx , aUseTheThuUmThe ; "Use the Thu'um - the Voice." .text: 000000000000 C9A6 mov rsi , rdx .text: 000000000000 C9A9 mov rdi , rax .text: 000000000000 C9AC call _choices_add .text: 000000000000 C9B1 .text: 000000000000 C9B1 loc_C9B1 While playing, I never managed to see the fourth choice. Obviously, there is a constraint checked in sub_C658 . We could quickly take a look at it (spoiler: it checks that the player's save file contains JDHACK ), but it would be better to avoid losing time; after all, it's a CTF, man! Let's just patch the library to force the fourth choice to be available: .text: 000000000000 C6A2 cmp dl , al .text: 000000000000 C6A4 jz short loc_C6AD .text: 000000000000 C6A6 mov eax , 1 ; we will replace this 1 by 0 .text: 000000000000 C6AB jmp short loc_C6BC With this patch, the hidden choice becomes available and brings us to an input box where we are asked to enter words: +------------------------------------------------------------------+ |Choose your words carefully Dovah | | > abcdefghijklm | +------------------------------------------------------------------+ Of course, if we enter some random content, we are returned that Language is Knowledge, Knowledge is Power. . It's worth noting that this string is not in the binary; maybe it's obfuscated or...