Systematic Debugging

Systematic Debugging

Overview

Debugging is investigation, not experimentation. This skill enforces a rigorous 4-phase process — root cause investigation, pattern analysis, hypothesis testing, and architecture questioning — that prevents shotgun debugging and ensures every fix is understood before it is applied.

Announce at start: "I'm using the systematic-debugging skill to investigate this issue."

Core Principle

┌─────────────────────────────────────────────────────────────────┐
│  HARD-GATE: NEVER GUESS. NEVER SHOTGUN DEBUG.                  │
│  NEVER CHANGE CODE WITHOUT UNDERSTANDING WHY IT IS BROKEN.     │
│                                                                 │
│  You are a detective gathering evidence, not a gambler trying   │
│  random fixes. If you are changing code without understanding   │
│  the root cause, STOP immediately.                             │
└─────────────────────────────────────────────────────────────────┘

Phase 1: Root Cause Investigation

Goal: Understand exactly WHAT is happening, not what you think is happening.

Actions

1. Read the error message carefully. The entire message. Every line. Including the stack trace.
2. Reproduce the bug. If you cannot reproduce it, you cannot fix it. Find the exact steps.
3. Gather evidence. Collect:
   • Full error message and stack trace
   • Input that triggers the bug
   • Expected behavior vs actual behavior
   • Environment details (versions, config, OS)
4. Check recent changes. What changed since this last worked?
   • Recent commits (git log, git diff)
   • Dependency updates
   • Configuration changes
   • Environment changes

Evidence Gathering Checklist

• [ ] Full error message captured (not truncated)
• [ ] Stack trace read from bottom to top
• [ ] Bug reproduced reliably with specific steps
• [ ] Expected vs actual behavior documented
• [ ] Recent changes reviewed (git log --oneline -20)
• [ ] Relevant logs examined

STOP — HARD-GATE: Do NOT proceed to Phase 2 until:

• [ ] You can reproduce the bug consistently
• [ ] You have the full error message and stack trace
• [ ] You know what changed recently
• [ ] You can describe the bug precisely (not vaguely)

Phase 2: Pattern Analysis

Goal: Narrow down WHERE the problem lives and WHEN it occurs.

Actions

1. Find working examples. Does this feature work in other contexts? With other inputs? In other environments?
2. Compare working vs broken. What is different between the case that works and the case that does not?
3. Check dependencies. Are all required services/libraries/configs present and correct?
4. Isolate the scope. Can you reproduce with a minimal example? Strip away everything non-essential.

Comparison Matrix

Fill this out to identify the pattern:

STOP — HARD-GATE: Do NOT proceed to Phase 3 until:

• [ ] You have identified at least one working case for comparison
• [ ] You have compared working vs broken and identified differences
• [ ] You have isolated the scope to the smallest reproducible case
• [ ] Dependencies have been verified (versions, availability, config)

Phase 3: Hypothesis and Testing

Goal: Form ONE specific, testable hypothesis and verify it with the smallest possible change.

Actions

1. Form ONE hypothesis. Based on evidence from Phases 1-2, what is the single most likely cause?
   • State it explicitly: "The bug occurs because [specific cause]"
   • If you cannot state it specifically, go back to Phase 1 or 2
2. Design a minimal test. What is the smallest change to confirm or deny this hypothesis?
   • Prefer adding a test case over modifying production code
   • Prefer logging/assertions over code changes
   • Prefer reverting a change over writing new code
3. Apply the change and test.
   • Make ONLY the change needed to test the hypothesis
   • Run the test suite
   • Observe the result
4. Evaluate.
   • If CONFIRMED: proceed with the fix, write a regression test
   • If DENIED: record what you learned, form a new hypothesis, return to step 1

Hypothesis Log Template

Hypothesis #1: [description]
Test: [what you did]
Result: CONFIRMED / DENIED
Learning: [what this taught you]

Hypothesis #2: ...

Decision Table: Hypothesis Testing Approach

STOP — HARD-GATE: Do NOT proceed to Phase 4 unless:

• [ ] You have tested at least 3 hypotheses and ALL were denied
• [ ] Each hypothesis was specific and testable
• [ ] Each test was minimal (one change at a time)
• [ ] You recorded learnings from each failed hypothesis

Phase 4: Architecture Questioning

Goal: If 3+ hypotheses have failed, the problem may be structural. Step back and question assumptions.

This phase is triggered ONLY after Phase 3 has been attempted at least 3 times without success.

Actions

1. Question your assumptions. What have you been assuming is true that might not be?
   • Is the data shaped the way you think it is?
   • Is the control flow what you expect?
   • Are the types what you think they are?
   • Is the API contract what you assumed?
2. Question the design. Is the current approach fundamentally flawed?
   • Is there a race condition in the design?
   • Is there a state management problem?
   • Is there an incorrect abstraction?
   • Are responsibilities misplaced?
3. Consider redesign. Sometimes the fix is not a patch but a restructuring.
   • Can you simplify the design to eliminate the bug class entirely?
   • Is there a pattern that handles this case better?
   • Should you replace rather than fix?
4. Seek external input. If you are stuck:
   • Explain the problem to someone else (rubber duck debugging)
   • Search for known issues in dependencies
   • Check if others have encountered similar problems

STOP — HARD-GATE: Do NOT continue without:

• [ ] Written list of assumptions that were questioned
• [ ] Explicit decision: patch the current design OR redesign
• [ ] If redesigning: a plan before implementing
• [ ] If patching: a new hypothesis informed by the assumption review

Debugging Decision Flowchart

Error encountered
    |
    v
Can you reproduce it?
    |
    +-- NO --> Gather more information (logs, user reports, monitoring)
    |          Try different inputs, environments, timing
    |          Do NOT proceed until reproducible
    |
    +-- YES -> Read the FULL error message and stack trace
               |
               v
         Is the cause obvious from the error?
               |
               +-- YES -> Form hypothesis, test it (Phase 3)
               |          Still write a regression test
               |
               +-- NO --> Complete Phase 1 evidence gathering
                          |
                          v
                    Find working case for comparison (Phase 2)
                          |
                          v
                    Identify differences
                          |
                          v
                    Form and test hypotheses (Phase 3)
                          |
                          +-- Fixed --> Write regression test, verify
                          |
                          +-- 3+ failed hypotheses --> Phase 4

Red Flags Table

Anti-Patterns / Common Mistakes

Integration Points

Quick Reference: What NOT To Do

1. Do NOT change random things and see if the bug goes away
2. Do NOT add try/catch to suppress the error
3. Do NOT rewrite the feature from scratch as a first resort
4. Do NOT blame the framework/library without evidence
5. Do NOT skip writing a regression test after fixing
6. Do NOT fix symptoms instead of root causes
7. Do NOT debug for more than 45 minutes without stepping back
8. Do NOT ignore error messages or stack traces

Skill Type

RIGID — The 4-phase process is mandatory and must be followed in order. Each phase has a HARD-GATE that must be satisfied before proceeding. Never change code without understanding why it is broken.