Reverse Engineering · Core Concepts & Methodology

A practical guide to understanding and modifying software without source code.

Prerequisites & Recommended Background

This material assumes you have completed at least one introductory programming course or have worked through a self‑study guide for C or C++ (such as the Teach Yourself X in 21 Days series). That background should be sufficient to follow the explanations and run the examples without difficulty.

However, additional preparation—especially in data structures (covering AVL trees, hash tables, graphs, and priority queues) and software engineering (particularly familiarity with design patterns)—will greatly assist you when you start making informed guesses and logical leaps. These topics are less about understanding the text itself and more about sharpening your analytical skills.

What Does “Reverse Engineering” Mean Here?

In the context of this book, reverse engineering is the process of determining how a piece of software—for which you have no source code—carries out a specific feature or function, to the point where you can either alter that code or replicate its behaviour in a separate project.

Full‑scale, ground‑up reverse engineering is notoriously difficult and often demands a team of engineers and specialised tooling just to capture the complete set of ideas within a system. But with the right utilities and a disciplined approach—especially keeping meticulous notes—you can extract the information you actually need: making modifications and introducing new behaviours into software that was never designed to allow them.

Why Bother with Reverse Engineering?

Because you can. It ultimately comes down to a desire for control and understanding. Enthusiasts, particularly in computing, are natural detail‑lovers. We enjoy grasping how a system works, predicting its every action, and even steering those actions. When you have source code, that’s straightforward. But when you don’t, reverse engineering becomes the only path.

Moreover, closed‑source software is often the most intriguing. You might be curious about a security mechanism, wonder whether copy protection truly is “unbreakable,” or simply want to know how a particular feature is implemented under the hood.

It Makes You a Better Programmer

This book will give you a deep, low‑level understanding of how your computer actually executes code. The more you know about that, the more efficient and robust your own programs will become.

Learning Assembly Language the Unconventional Way

If you don’t already know assembly, by the end of this book you will know it inside out. While most courses teach assembly as a standalone programming language, here you will learn to treat your C compiler as an assembly‑code generator. You’ll see how to view assembly through the lens of C, and how to think about C in terms of its generated machine instructions. This perspective gives you a significant edge—not only in programming skill but also in your ability to peer inside the “black box.” And you’ll have the satisfying answer to “Who taught you assembly?” — “My C compiler, of course!”

Legal Considerations

We will address legal issues in a later section. For now, be aware that shrink‑wrap licenses sometimes explicitly forbid reverse engineering, especially if you plan to publish your findings. Always check the applicable terms.

How to Approach This Book

Learn the General Methodology

This book covers reverse engineering on both UNIX (with a focus on GNU/Linux) and Microsoft Windows. Even if you currently work on only one platform, understanding both perspectives is valuable. You never know when a program on the “other” system will catch your interest, and having a broad mental framework helps you adapt quickly to new environments—whether that means 64‑bit architectures, new security models, or future paradigm shifts.

The core insight is to think about using tools and techniques to build as complete a “map” of your target as possible. Resist the temptation to treat any single tool or platform as the ultimate solution. Instead, focus on the process of gathering information, testing hypotheses, and piecing together the puzzle from multiple angles.

Read Between the Lines

We keep the text concise on purpose—there is a lot of ground to cover, and the learning is meant to be active, not passive. You won’t find exhaustive command‑by‑command summaries for every tool; many examples omit even the expected output. We assume you are either already comfortable with these utilities from everyday development or willing to experiment with them on your own.

That said, we do not skimp on the truly challenging material—assembly language, code modification techniques, and non‑trivial debugging methods. You are encouraged to follow along with your own projects and replicate the steps we describe.

Have a Clear Goal

None of this will stick unless you have a real reason to read. Pick a program and decide on a small, concrete piece of it that you want to understand. Perhaps you want to redirect a function call, replicate a feature in another context, intercept encrypted data before it leaves the machine, or even cheat in a multiplayer game. Having a tangible objective makes every lesson immediately relevant.

Keep a Notebook

Once you have your goal, organise your thoughts. Get a multi‑section notebook with areas for:

• General notes and observations
• Open questions
• Active hypotheses
• Experiments and results

Date every entry, and reserve one section for a chronological diary of what you have done. Every new fact you uncover is a small victory—write it down. Collecting this evidence pays off, especially in larger endeavours.

Use the Scientific Method

Yes, eighth‑grade science applies directly here. Reverse engineering is, in many ways, a proper science—arguably more so than much of the rest of computer science. Treat every target as a system to be studied. The method is an iterative cycle:

1. Observe and describe a phenomenon or set of phenomena. You notice something interesting—an unusual behaviour, a fluke, a sequence of events. Document it carefully, noting as many variables, prerequisites, and conditions as possible.
2. Formulate a hypothesis to explain what you observed. This should be an educated guess based on your understanding of software, data structures, and common patterns. Aim for a hypothesis that matches the complexity of the feature—if it likely fits in one function, it probably involves a handful of controlling variables. Predict what will happen if those variables change. You may also speculate about the underlying data structures or design patterns.
3. Test your hypothesis by predicting new events or—more often—by seeking evidence that disproves or refines it. The latter is especially useful early on, as it eliminates broad categories of possibility.
4. Use your insights to gain deeper understanding, and eventually to write code. Can you now predict how the system will behave under modified conditions? If you implement a component the way you believe the original works, will it perform the same job? For feature replication, try replacing the original code with your own version. For modifications, predict the result and verify it.

This is an iterative loop. Early passes deal with high‑level aspects and rely on simply using the application. Tools and assembly come in later iterations. The danger is skipping the hypothesis stage and jumping straight to testing—that yields only shallow results. Structure your tests to eliminate large families of explanations first, then home in on details.

If you have multiple hypotheses, design tests that discriminate among them. When a feature depends on many variables, organise them into a hierarchy—those with the biggest impact at the top. You can visualise this as a decision tree: a left branch might indicate a hypothesis is wrong, a right branch that it’s correct. A correct hypothesis often spawns a whole new subtree.

Start with an Aerial View, Then Zoom In

If an application is completely foreign or huge, begin with high‑level models. Data flow diagrams and activity diagrams are flexible tools that can represent a system at varying levels of abstraction. Choose the one that fits your thinking style—data flow is structural (C‑like), activity diagrams are more process‑oriented. These models help you identify major data paths and control flows before you ever look at a single instruction.

Book Structure Overview

The rest of the book gradually moves from general to specific. We first introduce tools that give you a whole‑system perspective—enough to form hypotheses about how your target accomplishes its tasks. Next, we use utilities that provide finer behavioural details. Then we re‑apply the scientific method to pinpoint the locations and purposes of interesting code segments, based on function calls and calling patterns. Finally, we dive into assembly to confirm or revise our understanding.

After that, we shift to applying this knowledge: code modification, function insertion, RPC interception, and buffer‑overflow techniques.

---

Chapter 2 – The Compilation Process

Compilation is normally split into five phases: preprocessing, parsing, translation, assembling, and linking. On UNIX, cc (or gcc) orchestrates all of them; you can see the individual steps with gcc -v. On Windows, the same phases exist under the MSVC++ front‑end (cl.exe), and the GNU toolchain is also available via MinGW or Cygwin. Cygwin provides a full POSIX layer, while MinGW offers just the build tools for native Windows binaries.

The Compiler Front‑End

• gcc – acts as a driver that invokes the preprocessor, compiler proper, assembler, and linker. Use gcc -v to see each sub‑command.
• cl.exe – the MSVC++ backend. Run cl -? for options. To use it from the command line, execute vsvars32.bat (found in the CommonX/Tools directory) to set the necessary environment variables.

Preprocessor

Handles all # directives in a single pass, performing macro expansion and file inclusion.

• gcc -E runs only the preprocessor; output goes to stdout (redirect with -o).
• cl -E does the same.

Parsing and Translation

This is the most instructive stage for our purposes because we can inspect the generated assembly. The UNIX and Windows worlds differ in their default assembly syntax (AT&T vs. Intel), but we will cover both. (You can also force Intel syntax in GCC with -masm=intel.)

• gcc -S produces a .s file in AT&T syntax; add -fverbose-asm to include helpful variable‑to‑stack annotations.
• Optimization levels are controlled with -O0 (none) through -O6 (maximum; often no further gains beyond -O4). Fine‑grained options like -funroll-loops, -finline-functions, and -fomit-frame-pointer can be turned on or off with their -fno- counterparts. Note that even at -O0, GCC 3.x may enable some optimisations; you can disable them individually if you want clearer output.
• On Windows, cl -S generates assembly and supports similar optimisations, though with fewer fine‑grained controls. Options like -Ob<n> (for inlining) and -Oy (omit frame pointer) are available.

Assembly Stage

The assembler converts assembly into machine code in an object file.

• GNU as accepts AT&T or Intel syntax and outputs .o files.
• Microsoft’s assembler is ml (MASM).

Linking

Both platforms support three linking styles:

• Static – library code is copied into the executable; function calls go directly to that code.
• Dynamic – the library is mapped once system‑wide and shared via virtual memory. Calls go through a Procedure Linkage Table (PLT) or jump table, which contains jumps to the dynamically resolved addresses.
• Runtime – the program explicitly loads a library with dlopen() (UNIX) or LoadLibrary() (Windows), then obtains function pointers via dlsym() or GetProcAddress(). This is common for plugin architectures.

• GNU linker ld (or collect2, as invoked by gcc) produces the final executable.
• Microsoft’s link.exe is used directly or via cl -link. Note that Windows requires a .lib or .def file alongside a .dll for static linking references.

---

Java Compilation and Execution

Java is a “semi‑interpreted” language. Source files (.java) are compiled to bytecode (.class files), which then runs inside the Java Virtual Machine (JVM). The compiler automatically compiles dependent classes if they are missing or out‑of‑date.

The JVM uses a class loader to locate and load bytecode. Developers can implement custom loaders that fetch classes from almost any source. The loading process calls loadClass(String name, boolean resolve), reads the bytecode, and passes it to defineClass. The bytecode verifier (invoked during defineClass) performs four passes to ensure safety. After verification, the class becomes available for execution.

Because Java bytecode is high‑level and retains much symbolic information, decompilation is straightforward—often recovering even variable and method names. Free tools like Jad (and others) can reconstruct readable source code from .class files. Consequently, this book does not dwell on reversing Java, as the bytecode is comparatively easy to analyse.

---