TL;DR
An 11.8 MB PE that is UPX over PyInstaller over 100 nested marshal layers over a Cython GUI over a custom VEH packer. The final gate is a 960-byte keyed comparison that inverts cleanly, and the last hop is a named-pipe service hidden in the UPX stub tail that <code>upx -d</code> throws away.
01Recon
Sample: Challenge.exe, 11,818,598 bytes, PE32+ x86-64.
Challenge.exe
|- UPX upx -d
|- PyInstaller 6.16.0 onefile overlay @0x6aa00, cookie @0xb4560e, 997 TOC entries
|- main.pyc 100 nested b85decode -> zlib -> marshal -> exec layers
| innermost: import enhanced; enhanced._c605a4bf()
|- enhanced.cp313-win_amd64.pyd Cython GUI + RPC client
|- check.cp313-win_amd64.pyd hand-written C behind the "UPYX" VEH packerenhanced.pyd is Cython built with CYTHON_COMPRESS_STRINGS plus a hand-rolled obfuscation layer: every arithmetic operator is a module-level lambda (so the decompilation contains no raw ^ & | * + - << >> %), every string literal is XOR-obfuscated with the fixed 17-byte keystream a69cf2432ece2c576eb85e17f8cd9bb8af, and every call goes through a dispatcher that resolves globals by a ror13 name hash, making the call graph invisible statically. All 45 obfuscated constants were decoded: GUI strings, kernel32 API names, and both pipe names \\.\pipe\supyxsvc and \\.\pipe\upyxsvc. No flag material.
“DON’T RENAME THIS FILE”
THE FILENAME WAS THE DECRYPTION KEY
02The bug
The hint is the key derivation. check.pyd walks PEB -> ProcessParameters -> CommandLine (gs:0x60 → [PEB+0x20] → Length at +0x70, Buffer at +0x78, CommandLine, not ImagePathName), takes the characters after the last., keeps alphanumerics, lowercases, and FNV-1a hashes them with a non- standard basis:
def fnv1a(s, basis=0xB5F6F899, prime=0x01000193): # basis is NOT 0x811C9DC5
h = basis
for c in s.encode(): h = ((h ^ c) * prime) & 0xFFFFFFFF
return h
K = fnv1a("exe") == 0x6716CD53 == 1729547603Rename Challenge.exe and K changes and nothing validates, "don't rename this file." The hint is in fact sharper than that: the key is whatever follows the last. on the command line, so running the module under python.exe -u script.py makes the extension "py".
The exploitable structure is in the verifier. check.check() writes {0x11} to a handle and reads 960 bytes, then runs a per-byte comparison whose transform is a bijection in the input byte for fixed context, and whose context uses only earlier bytes. That makes the required 960 bytes unique and invertible position by position.
03Exploitation
- Unwrap. upx -d, carve the PyInstaller CArchive (997 TOC entries), peel main.pyc's 100 nested exec(marshal.loads(zlib.decompress(base64.b85decode(BLOB)))) layers down to import enhanced; enhanced.
_c605a4bf(). - Decrypt
check.pyd. It is packed with a custom UPYX VEH packer, four encrypted regions decrypted on demand through a vectored exception handler. All four were decrypted statically. - Model the verifier.
FUN_18000e99e(buf960, const960, EXTKEY):
for i in 0..959:
prev = buf[i-1] if i >= 1 else 0x42
lag17 = buf[i-17] if i >= 17 else 0x99
lag73 = buf[i-73] if i >= 73 else 0xE7
if F(buf[i], prev, lag17, lag73, i, EXTKEY) != const[i]: return 0
return 1with const960 at.rdata VA 0x1800191f0 and
TAB = [3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59]
blk, pos = i >> 4, i & 15
k16 = (lag17 << 8) | prev
m16 = (lag73 << 8) | ((29*blk + 47*pos + 3*i) & 0xFF)
rot = (popcount(k16) + ctz(m16 | 0x8000) + blk + pos) & 0xF
v = rotl16(k16 ^ m16 ^ 0xA5C3, rot)
t = (PEXT(v,0x6DB6) << 1) ^ PEXT(v,0xB6D3) ^ popcount(v) ^ ctz(k16 | 0x100)
^ ((13*i + 0x5A) & 0xFF) ^ ((EXTKEY >> (8*(i & 3))) & 0xFF)
y = rotl8((x*TAB[pos] + ((11*blk + 17*pos + 0x71) & 0xFF)) & 0xFF, (blk+pos+i) & 7)
return y ^ (t & 0xFF)Every TAB[pos] is odd, so F is a bijection in x, invert it position by position.
- Invert.
required_960.bin, md5 0858d44140df33712e1cb7fb52f0c61c, unique, and verified four independent ways: analytic inversion; Unicorn emulation of the prologue with a synthetic PEB; executing the real binary (check.check()accepts any path, so a 961-byte file substitutes for the pipe, WriteFile overwrites byte 0 with0x11and ReadFile returns the 960 candidate bytes); and a model-free guard-page oracle, placing the buffer against PAGE_NOACCESS makes the correct byte the one that faults, since the loop only reads buf[i+1] if buf[i] matched. - The client model. The 960 bytes are 60 blocks of 16, one per typed character. Per keypress:
t = xorshift32((T[ord(ch)] * K + n) & 0xFFFFFFFF) # n = number of chars so far
r = rpc(b'\x01' + t.to_bytes(4,'little'), 16) # -> \\.\pipe\supyxsvc
if r is None or len(r) != 16: return
_6c1fa932.append(r) # appended VERBATIMSo the flag is exactly 60 characters, and block_n = g(t_n) where g is whatever the pipe service answers.
- Find the service. This is the part every static pass missed. The pipe names appear in the packed file only, upx -d discards them. In the tail of UPX1 sits an inserted native stub (~250 KB of obfuscated code, the two pipe names, a 480-byte blob at
0x5af20, then a jmp to the original entry point). RunningChallenge.exeunder wine + Xvfb and enumerating named pipes shows both pipes open, the service was inside the executable all along. - Query it. A small mingw bridge (CreateFileA/WriteFile/ReadFile/CloseHandle) turns the pipe into a scriptable oracle. It confirms
0x00→ seed, and that0x01+ t is a pure function of t (identical replies on repeat).recover_flag.pythen needs at most 60 x 65 = 3900 queries: for each position, try each on-screen character and keep the one whose 16-byte reply matchesrequired_960.bin[16n:16n+16].
04Dead ends
- EXTKEY is a hash, not a length. An earlier pass read the third argument of FUN_18000e99e as the length of the extension (3). It is EBX, the FNV-1a hash of the lowercased extension. This changed the required buffer from md5 a1778c... to 0858d4...; the superseded file is kept as
required_960_extlen3_WRONG.bin. tableT.jsonis a harness artifact, not the real S-box. It is _f0f17306(1), produced when an instrumentation stub replaced _aeb8ff4e with lambda: True. Every search was redone in an S-box- independent form (loop X over 0..255) as a result.- A reported 32-byte "hard-coded sentinel" in the validate routine was contamination, it was the value the instrumentation stub had itself fed in.
- Two ctypes function pointers were misidentified; resolved by address against the live export tables: _10cb7a81 = kernel32!FlsGetValue, _6db9f204 = ntdll!RtlGetCurrentPeb.
- FlsGetValue is declared restype=c_void_p, so a NULL slot yields None and the GUI aborts with "Something Wrong!", an anti-emulation gate, since under wine every FLS slot is NULL.
- Before the stub was found, an enormous search space was ruled out against the corrected 960 bytes: every hashlib algorithm over ~25 encodings of t; HMAC and keyed blake2 with every embedded constant; AES-128/192/256 in ECB/CBC/CTR both directions; ChaCha20, Salsa20, RC4, TEA, XTEA; 16 bytes from MT19937, xorshift32/64/128, splitmix, PCG and four LCGs; the UPYX packer's own keystream in 18 variants; a full 2^32 seed sweep over 16 PRNG families; 3,930 enumerated "small program" transforms; and 294 H(char, index) combinations. Zero hits, which was the correct signal that the transform was held by a service, not computable offline.
- Driving the real GUI: the button handler applies the Capslock flag to the button label, so with Capslock off the key labelled Q emits q. Feed the character, not the label.