When to Pivot

• - If you already understand the binary and now need heap, ROP, or kernel exploitation, switch to /ctf-pwn.
• If the challenge is really about recovering deleted files, PCAP data, or disk artifacts, switch to /ctf-forensics.
• If the target is a web app and you are only reversing a small client-side helper script, switch to /ctf-web.
• If the binary implements a machine learning model and the challenge is about model attacks or adversarial inputs, switch to /ctf-ai-ml.
• If the reversed binary's core logic is a cryptographic algorithm or math problem, switch to /ctf-crypto.
• If the binary is a real malware sample with C2, packing, or evasion behavior, switch to /ctf-malware.
• If the challenge is a toy VM, encoding puzzle, or pyjail rather than a real binary, switch to /ctf-misc.

Problem-Solving Workflow

1. 1. Start with strings extraction - many easy challenges have plaintext flags
2. Try ltrace/strace - dynamic analysis often reveals flags without reversing
3. Try Frida hooking - hook strcmp/memcmp to capture expected values without reversing
4. Try angr - symbolic execution solves many flag-checkers automatically
5. Try Qiling - emulate foreign-arch binaries or bypass heavy anti-debug without artifacts
6. Map control flow before modifying execution
7. Automate manual processes via scripting (r2pipe, Frida, angr, Python)
8. Validate assumptions by comparing decompiler outputs (dogbolt.org for side-by-side)

Quick Wins (Try First!)

CODEBLOCK4

Initial Analysis

CODEBLOCK5

Memory Dumping Strategy

Key insight: Let the program compute the answer, then dump it. Break at final comparison (b *main+OFFSET), enter any input of correct length, then x/s $rsi to dump computed flag.

Decoy Flag Detection

Pattern: Multiple fake targets before real check. Look for multiple comparison targets in sequence with different success messages. Set breakpoint at FINAL comparison, not earlier ones.

GDB PIE Debugging

PIE binaries randomize base address. Use relative breakpoints:
CODEBLOCK6

Comparison Direction (Critical!)

Two patterns: (1) transform(flag) == stored_target — reverse the transform. (2) transform(stored_target) == flag — flag IS the transformed data, just apply transform to stored target.

Common Encryption Patterns

• - XOR with single byte - try all 256 values
• XOR with known plaintext (flag{, CTF{)
• RC4 with hardcoded key
• Custom permutation + XOR
• XOR with position index (^ i or ^ (i & 0xff)) layered with a repeating key

Quick Tool Reference

CODEBLOCK7

Binary Types

Python .pyc

WASM

WASM game patching (Tac Tic Toe, Pragyan 2026): If proof generation is independent of move quality, patch minimax (flip i64.lt_s → i64.gt_s, change bestScore sign) to make AI play badly while proofs remain valid. Invoke /ctf-misc for full game patching patterns (games-and-vms).

Android APK

Flutter APK (Dart AOT)

.NET

• - dnSpy - debugging + decompilation
• ILSpy - decompiler

Packed (UPX)

upx -d packed -o unpacked

Tauri Packed Desktop Apps

Anti-Debugging Bypass

Common checks:

• - IsDebuggerPresent() / PEB.BeingDebugged / NtQueryInformationProcess (Windows)
• INLINECODE30 / /proc/self/status TracerPid (Linux)
• TLS callbacks (run before main — check PE TLS Directory)
• Timing checks (rdtsc, clock_gettime, GetTickCount)
• Hardware breakpoint detection (DR0-DR3 via GetThreadContext)
• INT3 scanning / code self-hashing (CRC over .text section)
• Signal-based: SIGTRAP handler, SIGALRM timeout, SIGSEGV for real logic
• Frida/DBI detection: /proc/self/maps scan, port 27042, inline hook checks

Bypass: Set breakpoint at check, modify register to bypass conditional.
pwntools patch: elf.asm(elf.symbols.ptrace, 'ret') to replace function with immediate return. See patterns.md.

For comprehensive anti-analysis techniques and bypasses (30+ methods with code), see anti-analysis.md.

S-Box / Keystream Patterns

Xorshift32: Shifts 13, 17, 5
Xorshift64: Shifts 12, 25, 27
Magic constants: 0x2545f4914f6cdd1d, INLINECODE38

Custom VM Analysis

1. 1. Identify structure: registers, memory, IP
2. Reverse executeIns for opcode meanings
3. Write disassembler mapping opcodes to mnemonics
4. Often easier to bruteforce than fully reverse
5. Look for the bytecode file loaded via command-line arg

See patterns.md for VM workflow, opcode tables, and state machine BFS.

Sequential key-chain brute-force: When a VM validates input in small blocks (e.g., 3 bytes = 2^24 candidates) with each block's output key feeding the next, brute-force each block sequentially with OpenMP parallelization. Compile solver with gcc -O3 -march=native -fopenmp. See patterns-ctf-3.md.

Python Bytecode Reversing

XOR flag checkers with interleaved even/odd tables are common. See languages.md for bytecode analysis tips and reversing patterns.

Signal-Based Binary Exploration

Binary uses UNIX signals as binary tree navigation; hook sigaction via LD_PRELOAD, DFS by sending signals. See patterns.md.

Malware Anti-Analysis Bypass via Patching

Flip JNZ/JZ (0x75/0x74), change sleep values, patch environment checks in Ghidra (Ctrl+Shift+G). See patterns.md.

Expected Values Tables

Locate with objdump -s -j .rodata binary | less — look near comparison instructions, size matches flag length.

x86-64 Gotchas

Sign extension and 32-bit truncation pitfalls. See patterns.md for details and code examples.

Iterative Solver Pattern

Try each byte (0-255) per position, match against expected output. Uniform transform shortcut: if one input byte only changes one output byte, build 0..255 mapping then invert. See patterns.md for full implementation.

Unicorn Emulation (Complex State)

INLINECODE47 -- map segments, set up stack, hook to trace. Mixed-mode pitfall: 64-bit stub jumping to 32-bit via retf requires switching to UCMODE32 and copying GPRs + EFLAGS + XMM regs. See tools.md.

Multi-Stage Shellcode Loaders

Nested shellcode with XOR decode loops; break at call rax, bypass ptrace with set $rax=0, extract flag from mov instructions. See patterns.md.

Timing Side-Channel Attack

Validation time varies per correct character; measure elapsed time per candidate to recover flag byte-by-byte. See patterns.md.

Godot Game Asset Extraction

Use KeyDot to extract encryption key from executable, then gdsdecomp to extract .pck package. See languages-platforms.md.

Roblox Place File Analysis

Query Asset Delivery API for version history; parse .rbxlbin chunks (INST/PROP/PRNT) to diff script sources across versions. See languages-platforms.md.

Unstripped Binary Information Leaks

Pattern: Debug info and file paths leak author identity. Quick checks: strings binary | grep "/home/" (home dirs), file binary (stripped?), readelf -S binary | grep debug (debug sections).

Custom Mangle Function Reversing

Binary mangles input 2 bytes at a time with running state; extract target from .rodata, write inverse function. See patterns.md.

Rust serde_json Schema Recovery

Disassemble serde Visitor implementations to recover expected JSON schema; field names in order reveal flag. See languages-platforms.md.

Position-Based Transformation Reversing

Binary adds/subtracts position index; reverse by undoing per-index offset. See patterns.md.

Hex-Encoded String Comparison

Input converted to hex, compared against constant. Decode with xxd -r -p. See patterns.md.

Embedded ZIP + XOR License Decryption

Binary with named symbols (EMBEDDED_ZIP, ENCRYPTED_MESSAGE) in .rodata → extract ZIP containing license, XOR encrypted message with license bytes to recover flag. No execution needed. See patterns-ctf-2.md.

Stack String Deobfuscation (.rodata XOR Blob)

Binary mmaps .rodata blob, XOR-deobfuscates, uses it to validate input. Reimplement verification loop with pyelftools to extract blob. Look for 0x9E3779B9, 0x85EBCA6B constants and rol32(). See patterns-ctf-2.md.

Prefix Hash Brute-Force

Binary hashes every prefix independently. Recover one character at a time by matching prefix hashes. See patterns-ctf-2.md.

Mathematical Convergence Bitmap

Pattern: Binary classifies coordinate pairs by Newton's method convergence (e.g., z^3-1=0). Grid of pass/fail results renders ASCII art flag. Key: the binary is a classifier, not a checker — reverse the math and visualize. See patterns-ctf.md.

RISC-V Binary Analysis

Statically linked, stripped RISC-V ELF. Use Capstone with CS_MODE_RISCVC | CS_MODE_RISCV64 for mixed compressed instructions. Emulate with qemu-riscv64. Watch for fake flags and XOR decryption with incremental keys. See tools.md.

Sprague-Grundy Game Theory Binary

Game binary plays bounded Nim with PRNG for losing-position moves. Identify game framework (Grundy values = pile % (k+1), XOR determines position), track PRNG state evolution through user input feedback. See patterns-ctf.md.

Kernel Module Maze Solving

Rust kernel module implements maze via device ioctls. Enumerate commands dynamically, build DFS solver with decoy avoidance, deploy as minimal static binary (raw syscalls, no libc). See patterns-ctf.md.

Multi-Threaded VM with Channels

Custom VM with 16+ threads communicating via futex channels. Trace data flow across thread boundaries, extract constants from GDB, watch for inverted validity logic, solve via BFS state space search. See patterns-ctf.md.

CVP/LLL Lattice for Constrained Integer Validation (HTB ShadowLabyrinth)

Binary validates flag via matrix multiplication with 64-bit coefficients; solutions must be printable ASCII. Use LLL reduction + CVP in SageMath to find nearest lattice point in the constrained range. Two-phase pattern: Phase 1 recovers AES key, Phase 2 decrypts custom VM bytecode with another linear system (mod 2^32). See patterns-ctf-2.md.

Decision Tree Function Obfuscation (HTB WonderSMS)

~200+ auto-generated functions routing input through polynomial comparisons. Script extraction via Ghidra headless rather than reversing each function manually. Constraint propagation from known output format cascades through arithmetic constraints. See patterns-ctf-2.md.

Android JNI RegisterNatives Obfuscation (HTB WonderSMS)

INLINECODE68 in JNI_OnLoad hides which C++ function handles each Java native method (no standard Java_com_pkg_Class_method symbol). Find the real handler by tracing JNI_OnLoad → RegisterNatives → fnPtr. Use x8664 .so from APK for best Ghidra decompilation. See languages-platforms.md.