Usama Arshad. Résumé

Challenge 11 of 15 Reversing Full writeup

BlackHat MEA 2026 · Qualifiers

upyx.

Are you familiar with python? If not, follow my friend’s advice, "don’t rename this file."

Reversing· 4 November 2026

TL;DR

An 11.8 MB PE that is UPX over PyInstaller over 100 nested marshal layers over a Cython GUI over a custom VEH packer. The final gate is a 960-byte keyed comparison that inverts cleanly, and the last hop is a named-pipe service hidden in the UPX stub tail that <code>upx -d</code> throws away.

Captured

01Recon

Sample: Challenge.exe, 11,818,598 bytes, PE32+ x86-64.

python
Challenge.exe
 |- UPX                            upx -d
     |- PyInstaller 6.16.0 onefile    overlay @0x6aa00, cookie @0xb4560e, 997 TOC entries
         |- main.pyc                  100 nested b85decode -> zlib -> marshal -> exec layers
         |                            innermost: import enhanced; enhanced._c605a4bf()
         |- enhanced.cp313-win_amd64.pyd   Cython GUI + RPC client
         |- check.cp313-win_amd64.pyd      hand-written C behind the "UPYX" VEH packer

enhanced.pyd is Cython built with CYTHON_COMPRESS_STRINGS plus a hand-rolled obfuscation layer: every arithmetic operator is a module-level lambda (so the decompilation contains no raw ^ & | * + - << >> %), every string literal is XOR-obfuscated with the fixed 17-byte keystream a69cf2432ece2c576eb85e17f8cd9bb8af, and every call goes through a dispatcher that resolves globals by a ror13 name hash, making the call graph invisible statically. All 45 obfuscated constants were decoded: GUI strings, kernel32 API names, and both pipe names \\.\pipe\supyxsvc and \\.\pipe\upyxsvc. No flag material.

“DON’T RENAME THIS FILE”

THE BRIEF 100 MARSHAL LAYERS A CYTHON RPC CLIENT A VEH PACKER AND A NAMED PIPE

THE FILENAME WAS THE DECRYPTION KEY

02The bug

The hint is the key derivation. check.pyd walks PEB -> ProcessParameters -> CommandLine (gs:0x60 → [PEB+0x20] → Length at +0x70, Buffer at +0x78, CommandLine, not ImagePathName), takes the characters after the last., keeps alphanumerics, lowercases, and FNV-1a hashes them with a non- standard basis:

python
def fnv1a(s, basis=0xB5F6F899, prime=0x01000193):   # basis is NOT 0x811C9DC5
    h = basis
    for c in s.encode(): h = ((h ^ c) * prime) & 0xFFFFFFFF
    return h

K = fnv1a("exe") == 0x6716CD53 == 1729547603

Rename Challenge.exe and K changes and nothing validates, "don't rename this file." The hint is in fact sharper than that: the key is whatever follows the last. on the command line, so running the module under python.exe -u script.py makes the extension "py".

The exploitable structure is in the verifier. check.check() writes {0x11} to a handle and reads 960 bytes, then runs a per-byte comparison whose transform is a bijection in the input byte for fixed context, and whose context uses only earlier bytes. That makes the required 960 bytes unique and invertible position by position.

03Exploitation

  1. Unwrap. upx -d, carve the PyInstaller CArchive (997 TOC entries), peel main.pyc's 100 nested exec(marshal.loads(zlib.decompress(base64.b85decode(BLOB)))) layers down to import enhanced; enhanced._c605a4bf().
  2. Decrypt check.pyd. It is packed with a custom UPYX VEH packer, four encrypted regions decrypted on demand through a vectored exception handler. All four were decrypted statically.
  3. Model the verifier.
snippet
FUN_18000e99e(buf960, const960, EXTKEY):
  for i in 0..959:
      prev = buf[i-1] if i >= 1 else 0x42
      lag17 = buf[i-17] if i >= 17 else 0x99
      lag73 = buf[i-73] if i >= 73 else 0xE7
      if F(buf[i], prev, lag17, lag73, i, EXTKEY) != const[i]: return 0
  return 1

with const960 at.rdata VA 0x1800191f0 and

snippet
TAB = [3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59]
blk, pos = i >> 4, i & 15
k16 = (lag17 << 8) | prev
m16 = (lag73 << 8) | ((29*blk + 47*pos + 3*i) & 0xFF)
rot = (popcount(k16) + ctz(m16 | 0x8000) + blk + pos) & 0xF
v   = rotl16(k16 ^ m16 ^ 0xA5C3, rot)
t   = (PEXT(v,0x6DB6) << 1) ^ PEXT(v,0xB6D3) ^ popcount(v) ^ ctz(k16 | 0x100)
      ^ ((13*i + 0x5A) & 0xFF) ^ ((EXTKEY >> (8*(i & 3))) & 0xFF)
y   = rotl8((x*TAB[pos] + ((11*blk + 17*pos + 0x71) & 0xFF)) & 0xFF, (blk+pos+i) & 7)
return y ^ (t & 0xFF)

Every TAB[pos] is odd, so F is a bijection in x, invert it position by position.

  1. Invert. required_960.bin, md5 0858d44140df33712e1cb7fb52f0c61c, unique, and verified four independent ways: analytic inversion; Unicorn emulation of the prologue with a synthetic PEB; executing the real binary (check.check() accepts any path, so a 961-byte file substitutes for the pipe, WriteFile overwrites byte 0 with 0x11 and ReadFile returns the 960 candidate bytes); and a model-free guard-page oracle, placing the buffer against PAGE_NOACCESS makes the correct byte the one that faults, since the loop only reads buf[i+1] if buf[i] matched.
  2. The client model. The 960 bytes are 60 blocks of 16, one per typed character. Per keypress:
snippet
t = xorshift32((T[ord(ch)] * K + n) & 0xFFFFFFFF)    # n = number of chars so far
r = rpc(b'\x01' + t.to_bytes(4,'little'), 16)       # -> \\.\pipe\supyxsvc
if r is None or len(r) != 16: return
_6c1fa932.append(r)                                 # appended VERBATIM

So the flag is exactly 60 characters, and block_n = g(t_n) where g is whatever the pipe service answers.

  1. Find the service. This is the part every static pass missed. The pipe names appear in the packed file only, upx -d discards them. In the tail of UPX1 sits an inserted native stub (~250 KB of obfuscated code, the two pipe names, a 480-byte blob at 0x5af20, then a jmp to the original entry point). Running Challenge.exe under wine + Xvfb and enumerating named pipes shows both pipes open, the service was inside the executable all along.
  2. Query it. A small mingw bridge (CreateFileA/WriteFile/ReadFile/CloseHandle) turns the pipe into a scriptable oracle. It confirms 0x00 → seed, and that 0x01 + t is a pure function of t (identical replies on repeat). recover_flag.py then needs at most 60 x 65 = 3900 queries: for each position, try each on-screen character and keep the one whose 16-byte reply matches required_960.bin[16n:16n+16].

04Dead ends

  • EXTKEY is a hash, not a length. An earlier pass read the third argument of FUN_18000e99e as the length of the extension (3). It is EBX, the FNV-1a hash of the lowercased extension. This changed the required buffer from md5 a1778c... to 0858d4...; the superseded file is kept as required_960_extlen3_WRONG.bin.
  • tableT.json is a harness artifact, not the real S-box. It is _f0f17306(1), produced when an instrumentation stub replaced _aeb8ff4e with lambda: True. Every search was redone in an S-box- independent form (loop X over 0..255) as a result.
  • A reported 32-byte "hard-coded sentinel" in the validate routine was contamination, it was the value the instrumentation stub had itself fed in.
  • Two ctypes function pointers were misidentified; resolved by address against the live export tables: _10cb7a81 = kernel32!FlsGetValue, _6db9f204 = ntdll!RtlGetCurrentPeb.
  • FlsGetValue is declared restype=c_void_p, so a NULL slot yields None and the GUI aborts with "Something Wrong!", an anti-emulation gate, since under wine every FLS slot is NULL.
  • Before the stub was found, an enormous search space was ruled out against the corrected 960 bytes: every hashlib algorithm over ~25 encodings of t; HMAC and keyed blake2 with every embedded constant; AES-128/192/256 in ECB/CBC/CTR both directions; ChaCha20, Salsa20, RC4, TEA, XTEA; 16 bytes from MT19937, xorshift32/64/128, splitmix, PCG and four LCGs; the UPYX packer's own keystream in 18 variants; a full 2^32 seed sweep over 16 PRNG families; 3,930 enumerated "small program" transforms; and 294 H(char, index) combinations. Zero hits, which was the correct signal that the transform was held by a service, not computable offline.
  • Driving the real GUI: the button handler applies the Capslock flag to the button label, so with Capslock off the key labelled Q emits q. Feed the character, not the label.