Run Test Plan

We are executing the test plan. All implementation is complete. Now we verify it works.

nathanonn 2 updated 3mo ago
Claude CodeGeneric
View source ↗
We are executing the test plan. All implementation is complete. Now we verify it works.

## Reference Documents

- **Test Plan:** `notes/test_plan.md`
- **Implementation Plan:** `notes/impl_plan.md`
- **Specs:** `notes/specs.md`
- **Test Status JSON:** `notes/test-status.json`
- **Test Results Log:** `notes/test-results.md`

---

## Phase 1: Initialize

### Step 1: Check for Existing Run (Resumption)

Before creating anything, check if a previous test run exists:

1. Check if `notes/test-status.json` exists
2. Check if there are existing tasks via `TaskList`

**If both exist and tasks have results:**

- This is a **resumed run** — skip to Phase 2 (Step 7)
- Announce: "Resuming previous test run. Skipping already-passed tests."
- Only execute tasks that are still `pending` or `fail` (with fixAttempts < 3)

**If no previous run exists (or files are missing):**

- Continue with fresh initialization below

### Step 2: Read the Test Plan

Read `notes/test_plan.md` and extract ALL test cases. Auto-detect the TC-ID pattern used (e.g., `TC-001`, `TC-101`, `TC-5A`, etc.).

For each test case, note:

- TC ID
- Name
- Priority (Critical / High / Medium / Low — default to Medium if not stated)
- Preconditions
- Test steps and expected outcomes
- Test data (if any)
- Dependencies on other test cases (if any)

### Step 3: Analyze Test Dependencies

Determine which test cases depend on others. Common dependency patterns:

- A "saves data" test may depend on a "displays default" test
- A "form submission" test may depend on "form validation" tests
- An "end-to-end" test may depend on individual component tests

If no clear dependencies exist between test cases, treat them all as independent.

### Step 4: Create Tasks

Use `TaskCreate` to create one task per test case. Set `blocked_by` based on the dependency analysis.

**Task description format:**

Test [TC-ID]: [Test Name] Priority: [Priority]

Preconditions:

  • [Required state before test]

Steps:

Step Action Expected Result
1 [Action] [Result]
2 [Action] [Result]

Test Data:

Expected Outcome: [Final verification]


fixAttempts: 0 result: pending lastTestedAt: null notes:


### Step 5: Generate Test Status JSON

Create `notes/test-status.json`:

```json
{
    "metadata": {
        "testPlanSource": "notes/test_plan.md",
        "totalIterations": 0,
        "maxIterations": 50,
        "startedAt": null,
        "lastUpdatedAt": null,
        "summary": {
            "total": "<count>",
            "pending": "<count>",
            "pass": 0,
            "fail": 0,
            "knownIssue": 0
        }
    },
    "testCases": {
        "<TC-ID>": {
            "name": "Test case name",
            "priority": "Critical|High|Medium|Low",
            "status": "pending",
            "fixAttempts": 0,
            "notes": "",
            "lastTestedAt": null
        }
    },
    "knownIssues": []
}

Step 6: Initialize Test Results Log

Create notes/test-results.md:

# Test Results

**Test Plan:** notes/test_plan.md
**Started:** [CURRENT_TIMESTAMP]

## Execution Log

Verify Initialization

Use TaskList to confirm:

  • All TC-IDs from the test plan have a corresponding task
  • Dependencies are correctly set via blocked_by
  • All tasks show result: pending

Cross-check task count matches summary.total in notes/test-status.json.


Phase 2: Execute Tests

Step 7: Determine Execution Order

Use TaskList to read all tasks and their blocked_by fields. Determine sequential execution order:

  1. Tasks with no blocked_by (or all dependencies resolved) come first
  2. Tasks whose dependencies are resolved come next
  3. Continue until all tasks are ordered

For resumed runs: Skip tasks where result is already pass or known_issue.

Step 8: Execute One Task at a Time

For the next eligible task, spawn ONE sub-agent with the instructions below.

One sub-agent at a time. Do NOT spawn multiple sub-agents in parallel.


Sub-Agent Instructions

You are a test execution sub-agent. You have ONE job: execute and verify ONE test case.

  1. Read your task using TaskGet to get the full description

  2. Parse the test steps from the description (everything above the --- separator)

  3. Parse the metadata from below the --- separator

  4. Read CLAUDE.md for environment details, URLs, and credentials

  5. Execute the test:

    Using browser automation:

    • Navigate to URLs specified in the test steps
    • Click buttons/links as described
    • Fill form inputs with the test data provided
    • Take screenshots at key verification points
    • Read console logs for errors
    • Verify DOM state matches expected outcomes

    Follow the test plan steps EXACTLY. Do not skip steps.

  6. Determine the result:

    PASS if:

    • All expected outcomes verified
    • No unexpected console errors
    • UI state matches test plan

    FAIL if:

    • Any expected outcome not met
    • Unexpected errors
    • UI state doesn't match
  7. If PASS: Update the task description metadata via TaskUpdate:

    ---
    fixAttempts: 0
    result: pass
    lastTestedAt: [CURRENT_TIMESTAMP]
    notes: [Brief description of what was verified]
    

    Mark the task as completed.

  8. If FAIL and fixAttempts < 3:

    a. Analyze the root cause b. Implement a fix in the codebase c. Increment fixAttempts and update via TaskUpdate:

    ---
    fixAttempts: [previous + 1]
    result: fail
    lastTestedAt: [CURRENT_TIMESTAMP]
    notes: [What failed, root cause, what fix was applied]
    

    d. Re-run the test steps to verify the fix e. If now passing, set result: pass and mark task as completed f. If still failing and fixAttempts < 3, repeat from (a)

  9. If FAIL and fixAttempts >= 3: Mark as known issue via TaskUpdate:

    ---
    fixAttem
    

Maintain Run Test Plan?

Let people know it's listed here — add the badge (live metrics, light/dark aware) or a plain link to your README or docs.

[Run Test Plan on getagentictools](https://getagentictools.com/loops/nathanonn-test-results?ref=badge)