Devops Check

---
description: Monitor deployments and fix failures automatically by launching parallel monitoring agents
allowed-tools: Bash(git:*), Bash(gh:*), Bash(aws:*), Task(*)
model: sonnet
---

# DevOps Check Command

Launches specialized monitoring agents to watch deployments and automatically fixes any failures that occur. This command is designed to handle long-running deployments (10+ minutes) and process failure logs to implement fixes.

## Overview

This command:
1. **Detects** what deployment systems are configured (GitHub Actions, AWS CodePipeline)
2. **Launches** parallel monitoring agents for each system
3. **Monitors temporal window** - For GitHub Actions, monitors all workflows from the last 5 minutes
4. **Tracks cascading workflows** - Waits for and monitors workflows triggered by other workflows (e.g., Test → Deploy chains)
5. **Waits** for all deployments to complete (even if they take 10+ minutes per workflow)
6. **Collects** failure logs from any failed deployments
7. **Fixes** issues automatically based on the logs
8. **Re-monitors** automatically after applying fixes with a fresh monitoring window
9. **Iterates** until all pipelines pass (up to 5 attempts)

**Key Features:**
- **Comprehensive monitoring**: Doesn't just check the latest action, monitors ALL recent workflows
- **Cascade detection**: Automatically detects and waits for workflows triggered by other workflows
- **Temporal boundary**: Monitors workflows from the last 5 minutes + any future cascading workflows
- **Example**: If a "Test" workflow succeeds and triggers a "Deploy" workflow, both are monitored and reported

## Workflow

### Step 1: Detect Current Branch and Repository Info

Determine the current deployment context:

```bash
# Get current branch
CURRENT_BRANCH=$(git branch --show-current)

# Get repository name from package.json (remove scope if exists)
REPO_NAME=$(node -p "const name = require('./package.json').name; name.startsWith('@') ? name.split('/')[1] : name")

# Construct AWS CLI profile name: <repo-name>-<environment>
AWS_PROFILE="${REPO_NAME}-${CURRENT_BRANCH}"

# Examples:
# - Repository: bws-backoffice, Branch: staging → Profile: bws-backoffice-staging
# - Repository: bws-api, Branch: prod → Profile: bws-api-prod

AWS Profile Naming Pattern:

• Format: <repo-name>-<environment>
• Repository name is extracted from package.json (scope removed)
• Environment is the current branch name
• Examples:
  • bws-backoffice-staging
  • bws-backoffice-prod
  • bws-api-staging

Step 2: Detect Configured Systems

Check what deployment systems are configured:

GitHub Actions:

# Check if .github/workflows/ exists
if [ -d ".github/workflows" ]; then
    echo "GitHub Actions detected"
fi

AWS CodePipeline:

# Check if .worktree-config.json indicates AWS is used
# Or check if .deploy/ directory exists
if [ -f ".worktree-config.json" ]; then
    USE_AWS=$(jq -r '.config.useAWS' .worktree-config.json)
fi

# Also check for .deploy/IaC directory
if [ -d ".deploy/IaC" ]; then
    USE_AWS="true"
fi

Step 3: Launch Monitoring Agents in Parallel

IMPORTANT: Launch agents using the Task tool with multiple calls in a single message to run them in parallel.

Create monitoring agents based on what's configured:

Agent 1: GitHub Actions Monitor

Task agent with the following prompt:

You are a GitHub Actions monitoring agent. Your job is to monitor ALL GitHub Actions workflow runs within a time window and wait for cascading workflows.

**Branch:** {CURRENT_BRANCH}
**Repository:** {OWNER}/{REPO}
**Monitoring Window:** Last 5 minutes + future cascading workflows

**Your tasks:**

**Step 1: Define Temporal Boundary**

Get the timestamp for "5 minutes ago" to establish the monitoring window:
```bash
# Get timestamp for 5 minutes ago (ISO 8601 format)
MONITORING_START=$(date -u -d '5 minutes ago' '+%Y-%m-%dT%H:%M:%SZ')
echo "Monitoring workflows since: $MONITORING_START"

Step 2: Get ALL Recent Workflow Runs

Get all workflow runs that started within the monitoring window:

# Get all runs from the last 5 minutes for this branch
gh run list --branch {CURRENT_BRANCH} --limit 50 \
  --json databaseId,status,conclusion,name,workflowName,createdAt \
  --jq "[.[] | select(.createdAt >= \"$MONITORING_START\")]"

IMPORTANT: This may return multiple workflows:

• Initial workflow (e.g., "Test")
• Cascading workflows (e.g., "Deploy" triggered by "Test" success)
• Re-runs or multiple pushes

Step 3: Monitor All Workflows Until Completion

For each workflow run found:

1. Track its status
2. Wait for completion
3. After completion, check if new workflows were triggered

# For each run ID from Step 2:
for RUN_ID in {RUN_IDS}; do
    echo "Monitoring workflow run: $RUN_ID"

    # Watch the run (this command waits for completion)
    gh run watch $RUN_ID

    echo "Workflow $RUN_ID completed"
done

IMPORTANT: This monitoring may take 10+ minutes per workflow.

Step 4: Check for Newly Triggered Workflows

After workflows complete, check if any new workflows were triggered:

# Wait 30 seconds for any cascade triggers
sleep 30

# Get current timestamp
CURRENT_TIME=$(date -u '+%Y-%m-%dT%H:%M:%SZ')

# Check for new workflows created after our last check
gh run list --branch {CURRENT_BRANCH} --limit 20 \
  --json databaseId,status,conclusion,name,workflowName,createdAt \
  --jq "[.[] | select(.createdAt >= \"$LAST_CHECK_TIME\")]"

If new workflows found:

• Add them to monitoring list
• Repeat Step 3 for new workflows
• Continue checking for cascades until no new workflows appear for 1 minute

Step 5: Collect Results from All Workflows

After all workflows complete and no new ones are triggered:

# For each monitored run:
for RUN_ID in {ALL_RUN_IDS}; do
    echo "Getting results for run: $RUN_ID"

    # Get final status
    gh run view $RUN_ID --json conclusion,status,jobs,name,workflowName

    # If fa