cicd-intelligent-recovery
Loop 3 of the Three-Loop Integrated Development System. CI/CD automation with intelligent failure recovery, root cause analysis, and comprehensive quality validation. Receives implementation from Loop 2, feeds failure patterns back to Loop 1. Achieves 100% test success through automated repair and theater validation. v2.0.0 with explicit agent SOPs.
- Category: discovered
- Tags: github, community
- Version: 1.0.0
- Author: walid
- Downloads: 0
- Updated: 2025-11-21
Download the latest version (v1.0.0) from this page.
SKILL.md
---
name: cicd-intelligent-recovery
description: Loop 3 of the Three-Loop Integrated Development System. CI/CD automation with intelligent failure recovery, root cause analysis, and comprehensive quality validation. Receives implementation from Loop 2, feeds failure patterns back to Loop 1. Achieves 100% test success through automated repair and theater validation. v2.0.0 with explicit agent SOPs.
---
# CI/CD Quality & Debugging Loop (Loop 3)
**Purpose**: Continuous integration with automated failure recovery and authentic quality validation.
**SOP Workflow**: Specification → Research → Planning → Execution → Knowledge
**Output**: 100% test success rate with authentic quality improvements and failure pattern analysis
**Integration**: This is Loop 3 of 3. Receives from `parallel-swarm-implementation` (Loop 2), feeds failure data back to `research-driven-planning` (Loop 1).
**Version**: 2.0.0
**Optimization**: Evidence-based prompting with explicit agent SOPs
---
## When to Use This Skill
Activate this skill when:
- Have complete implementation from Loop 2 (parallel-swarm-implementation)
- Need CI/CD pipeline automation with intelligent recovery
- Require root cause analysis for test failures
- Want automated repair with connascence-aware fixes
- Need validation of authentic quality (no theater)
- Generating failure patterns for Loop 1 feedback
**DO NOT** use this skill for:
- Initial development (use Loop 2 first)
- Manual debugging without CI/CD integration
- Quality checks during development (use Loop 2 theater detection)
---
## Input/Output Contracts
### Input Requirements
```yaml
input:
loop2_delivery_package:
location: .claude/.artifacts/loop2-delivery-package.json
schema:
implementation: object (complete codebase)
tests: object (test suite)
theater_baseline: object (theater metrics from Loop 2)
integration_points: array[string]
validation:
- Must exist and be valid JSON
- Must include theater_baseline for differential analysis
ci_cd_failures:
source: GitHub Actions workflow runs
format: JSON array of failure objects
required_fields: [file, line, column, testName, errorMessage, runId]
github_credentials:
required: gh CLI authenticated
check: gh auth status
```
### Output Guarantees
```yaml
output:
test_success_rate: 100% (guaranteed)
quality_validation:
theater_audit: PASSED (no false improvements)
sandbox_validation: 100% test pass
differential_analysis: improvement metrics
failure_patterns:
location: .claude/.artifacts/loop3-failure-patterns.json
feeds_to: Loop 1 (next iteration)
schema:
patterns: array[failure_pattern]
recommendations: object (planning/architecture/testing)
delivery_package:
location: .claude/.artifacts/loop3-delivery-package.json
contains:
- quality metrics (test success, failures fixed)
- analysis data (root causes, connascence context)
- validation results (theater, sandbox, differential)
- feedback for Loop 1
```
---
## Prerequisites
Before starting Loop 3, ensure Loop 2 completion:
```bash
# Verify Loop 2 delivery package exists
test -f .claude/.artifacts/loop2-delivery-package.json && echo "✅ Ready" || echo "❌ Run parallel-swarm-implementation first"
# Load implementation data
npx claude-flow@alpha memory query "loop2_complete" --namespace "integration/loop2-to-loop3"
# Verify GitHub CLI authenticated
gh auth status || gh auth login
```
---
## 8-Step CI/CD Process Overview
```
Step 1: GitHub Hook Integration (Download CI/CD failure reports)
↓
Step 2: AI-Powered Analysis (Gemini + 7-agent synthesis with Byzantine consensus)
↓
Step 3: Root Cause Detection (Graph analysis + Raft consensus)
↓
Step 4: Intelligent Fixes (Program-of-thought: Plan → Execute → Validate → Approve)
↓
Step 5: Theater Detection Audit (6-agent Byzantine consensus validation)
↓
Step 6: Sandbox Validation (Isolated production-like testing)
↓
Step 7: Differential Analysis (Compare to baseline with metrics)
↓
Step 8: GitHub Feedback (Automated reporting and loop closure)
```
---
## Step 1: GitHub Hook Integration
**Objective**: Download and process CI/CD pipeline failure reports from GitHub Actions.
**Agent Coordination**: Single orchestrator agent manages data collection.
### Configure GitHub Hooks
```bash
# Install GitHub CLI if needed
which gh || brew install gh
# Authenticate
gh auth login
# Configure webhook listener
gh api repos/{owner}/{repo}/hooks \
-X POST \
-f name='web' \
-f active=true \
-f events='["check_run", "workflow_run"]' \
-f config[url]='http://localhost:3000/hooks/github' \
-f config[content_type]='application/json'
```
### Download Failure Reports
```bash
# Get recent workflow runs
gh run list --repo {owner}/{repo} --limit 10 --json conclusion,databaseId \
| jq '.[] | select(.conclusion == "failure")' \
> .claude/.artifacts/failed-runs.json
# Download logs for each failure
cat .claude/.artifacts/failed-runs.json | jq -r '.databaseId' | while read RUN_ID; do
gh run view $RUN_ID --log \
> .claude/.artifacts/failure-logs-$RUN_ID.txt
done
```
### Parse Failure Data
```bash
node <<'EOF'
const fs = require('fs');
const failures = [];
// Parse all failure logs
const logFiles = fs.readdirSync('.claude/.artifacts')
.filter(f => f.startsWith('failure-logs-'));
logFiles.forEach(file => {
const log = fs.readFileSync(`.claude/.artifacts/${file}`, 'utf8');
// Extract structured failure data
const failureMatches = log.matchAll(/FAIL (.+?):(\d+):(\d+)\n(.+?)\n(.+)/g);
for (const match of failureMatches) {
failures.push({
file: match[1],
line: parseInt(match[2]),
column: parseInt(match[3]),
testName: match[4],
errorMessage: match[5],
runId: file.match(/failure-logs-(\d+)/)[1]
});
}
});
fs.writeFileSync(
'.claude/.artifacts/parsed-failures.json',
JSON.stringify(failures, null, 2)
);
console.log(`✅ Parsed ${failures.length} failures`);
EOF
```
**Validation Checkpoint**:
- ✅ Failure data parsed and structured
- ✅ All required fields present (file, line, testName, errorMessage)
---
## Step 2: AI-Powered Analysis
**Objective**: Use Gemini large-context analysis + 7 research agents with Byzantine consensus to examine each failure deeply.
**Evidence-Based Techniques**: Self-consistency, Byzantine consensus, program-of-thought
### Phase 1: Gemini Large-Context Analysis
**Leverage Gemini's 2M token window for full codebase analysis**
```bash
# Analyze failures with full codebase context
/gemini:impact "Analyze CI/CD test failures:
FAILURE DATA:
$(cat .claude/.artifacts/parsed-failures.json)
CODEBASE CONTEXT:
Full repository (all files)
LOOP 2 IMPLEMENTATION:
$(cat .claude/.artifacts/loop2-delivery-package.json)
ANALYSIS OBJECTIVES:
1. Identify cross-file dependencies related to failures
2. Detect failure cascade patterns (root → secondary → tertiary)
3. Analyze what changed between working and failing states
4. Assess system-level architectural impact
5. Identify connascence patterns in failing code
OUTPUT FORMAT:
{
dependency_graph: { nodes: [files], edges: [dependencies] },
cascade_map: { root_failures: [], cascaded_failures: [] },
change_analysis: { changed_files: [], change_impact: [] },
architectural_impact: { affected_systems: [], coupling_issues: [] }
}"
# Store Gemini analysis
cat .claude/.artifacts/gemini-response.json \
> .claude/.artifacts/gemini-analysis.json
```
### Phase 2: Parallel Multi-Agent Deep Dive (Self-Consistency)
**7 parallel agents for cross-validation and consensus**
```javascript
// PARALLEL ANALYSIS AGENTS - Evidence-Based Self-Consistency
[Single Message - Spawn All 7 Analysis Agents]:
// Failure Pattern Research (Dual agents for cross-validation)
Task("Failure Pattern Researcher 1",
`Research similar failures in external sources:
- GitHub issues for libraries we use
- Stack Overflow questions with similar error messages
- Documentation of known issues
Failures to research: $(cat .claude/.artifacts/parsed-failures.json | jq -r '.[].errorMessage')
For each failure:
1. Find similar reported issues
2. Document known solutions with evidence (links, code examples)
3. Note confidence level (high/medium/low)
Store findings: .claude/.artifacts/failure-patterns-researcher1.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "research-patterns-1"`,
"researcher")
Task("Failure Pattern Researcher 2",
`Cross-validate findings from Researcher 1:
- Load: .claude/.artifacts/failure-patterns-researcher1.json
- Verify each claimed solution independently
- Check for conflicting solutions
- Identify most reliable approaches
For conflicts:
1. Research both approaches
2. Determine which is more current/reliable
3. Flag disagreements for consensus
Store findings: .claude/.artifacts/failure-patterns-researcher2.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "research-patterns-2"`,
"researcher")
// Error Analysis Specialist
Task("Error Message Analyzer",
`Deep dive into error messages and stack traces:
Failures: $(cat .claude/.artifacts/parsed-failures.json)
For each error message:
1. Parse error semantics (syntax error vs runtime vs logic)
2. Extract root cause from stack trace (not just symptoms)
3. Identify error propagation patterns
4. Distinguish between:
- Direct causes (code that threw error)
- Indirect causes (code that set up failure conditions)
Apply program-of-thought reasoning:
"Error X occurred because Y, which was caused by Z"
Store analysis: .claude/.artifacts/error-analysis.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "error-analysis"`,
"analyst")
// Code Context Investigator
Task("Code Context Investigator",
`Analyze surrounding code context for failures:
Load: .claude/.artifacts/parsed-failures.json
Load: .claude/.artifacts/gemini-analysis.json (for dependency context)
For each failure:
1. Read file at failure line ±50 lines
2. Identify why failure occurs in THIS specific codebase
3. Find coupling issues (tight coupling → cascading failures)
4. Analyze code smells that contributed to failure
Context analysis:
- Variable/function naming clarity
- Error handling presence/absence
- Input validation
- Edge case handling
Store findings: .claude/.artifacts/code-context.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "code-context"`,
"code-analyzer")
// Test Validity Auditors (Dual agents for critical validation)
Task("Test Validity Auditor 1",
`Determine if tests are correctly written:
Load failures: .claude/.artifacts/parsed-failures.json
For each failing test:
1. Is test logic correct? (proper assertions, valid test data)
2. Is failure indicating real bug or test issue?
3. Check test quality:
- Proper setup/teardown
- Isolated (not depending on other tests)
- Deterministic (not flaky)
Categorize:
- Real bugs: Code is wrong, test is correct
- Test issues: Code is correct, test is wrong
- Both wrong: Code and test both have issues
Store analysis: .claude/.artifacts/test-validity-1.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "test-validity-1"`,
"tester")
Task("Test Validity Auditor 2",
`Cross-validate test analysis from Auditor 1:
Load: .claude/.artifacts/test-validity-1.json
Load: .claude/.artifacts/loop2-delivery-package.json (theater baseline)
Additional checks:
1. Compare to Loop 2 theater baseline
2. Check for test theater patterns:
- Meaningless assertions (expect(1).toBe(1))
- Over-mocking (mocking the thing being tested)
- False positives (tests that don't actually test)
For disagreements with Auditor 1:
1. Re-examine test thoroughly
2. Document reasoning for different conclusion
3. Flag for consensus resolution
Store analysis: .claude/.artifacts/test-validity-2.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "test-validity-2"`,
"tester")
// Dependency Specialist
Task("Dependency Conflict Detector",
`Check for dependency-related failures:
Load failures: .claude/.artifacts/parsed-failures.json
Analysis steps:
1. Check package.json/requirements.txt for version conflicts
2. Identify breaking changes in dependencies:
- Compare current versions to last working versions
- Review CHANGELOG files for breaking changes
- Check deprecation warnings
3. Analyze transitive dependencies:
- npm ls or pip list --tree
- Find version conflicts in dep tree
4. Check for missing dependencies:
- ImportError / Cannot find module
- Missing peer dependencies
Store findings: .claude/.artifacts/dependency-analysis.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "dependency-analysis"`,
"analyst")
// Wait for all 7 agents to complete
npx claude-flow@alpha task wait --all --namespace "cicd/analysis"
```
### Phase 3: Synthesis with Byzantine Consensus
**Byzantine fault-tolerant synthesis requires 5/7 agent agreement**
```javascript
[Single Message - Synthesis Coordinator]:
Task("Analysis Synthesis Coordinator",
`Synthesize findings from Gemini + 7 agents using Byzantine consensus.
INPUTS:
- Gemini Analysis: .claude/.artifacts/gemini-analysis.json
- Researcher 1: .claude/.artifacts/failure-patterns-researcher1.json
- Researcher 2: .claude/.artifacts/failure-patterns-researcher2.json
- Error Analyzer: .claude/.artifacts/error-analysis.json
- Code Context: .claude/.artifacts/code-context.json
- Test Auditor 1: .claude/.artifacts/test-validity-1.json
- Test Auditor 2: .claude/.artifacts/test-validity-2.json
- Dependency Detector: .claude/.artifacts/dependency-analysis.json
SYNTHESIS PROCESS:
1. Cross-Reference Analysis:
For each failure, collect all agent findings
Build confidence matrix: which agents agree on root cause
2. Byzantine Consensus:
For each root cause claim:
- Count agent agreement (need 5/7 for consensus)
- Weight by agent confidence scores
- Flag conflicts (< 5/7 agreement) for manual review
3. Consolidate Root Causes:
- Primary causes: 7/7 agreement (highest confidence)
- Secondary causes: 5-6/7 agreement (medium confidence)
- Disputed causes: < 5/7 agreement (flag for review)
4. Generate Synthesis Report:
{
rootCauses: [
{
failure: failure_object,
cause: "root cause description",
evidence: ["agent1 finding", "agent2 finding"],
consensus: 7/7 or 6/7 or 5/7,
confidence: "high" | "medium" | "low"
}
],
cascadingFailures: [
{ root: failure_id, cascaded: [failure_ids] }
],
quickWins: [ /* easy fixes */ ],
complexIssues: [ /* require architecture changes */ ]
}
VALIDATION:
- All failures must be categorized
- Root causes must have >= 5/7 consensus or be flagged
- Cascading relationships must be validated by Gemini graph
Store: .claude/.artifacts/analysis-synthesis.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "synthesis-consensus"`,
"byzantine-coordinator")
```
**Validation Checkpoint**:
- ✅ Gemini analysis complete (dependency graph, cascade map)
- ✅ All 7 agents completed analysis
- ✅ Byzantine consensus achieved (5/7 agreement on root causes)
- ✅ Synthesis report generated with confidence scores
---
## Step 3: Root Cause Detection
**Objective**: Reverse engineer to find cascade issues and true root causes using graph analysis and Raft consensus.
**Evidence-Based Techniques**: Graph algorithms, connascence analysis, Raft consensus
### Phase 1: Parallel Cascade Graph Analysis
**Multiple graph analysts for validation**
```javascript
[Single Message - Parallel Graph Analysis]:
Task("Failure Graph Analyst 1",
`Build failure dependency graph using graph algorithms:
Load: .claude/.artifacts/parsed-failures.json
Load: .claude/.artifacts/gemini-analysis.json (dependency context)
GRAPH CONSTRUCTION:
1. Nodes: Each failure is a node
2. Edges: Failure A → Failure B if:
- B's error message references A's file
- B's file imports A's file
- B's line number > A's line number in same file
- Gemini dependency graph shows A → B relationship
3. Apply graph algorithms:
- Topological sort to find root nodes (no incoming edges)
- Calculate cascade depth (max distance from root)
- Find strongly connected components (circular dependencies)
4. Identify root causes:
Root = node with 0 incoming edges OR
Root = node in cycle with most outgoing edges
OUTPUT:
{
graph: { nodes: [], edges: [] },
roots: [ /* root failure nodes */ ],
cascadeMap: { /* depth levels */ },
circularDeps: [ /* cycles detected */ ]
}
Store: .claude/.artifacts/failure-graph-1.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "graph-1"`,
"analyst")
Task("Failure Graph Analyst 2",
`Validate graph structure from Analyst 1:
Load: .claude/.artifacts/failure-graph-1.json
Load: .claude/.artifacts/analysis-synthesis.json (consensus data)
VALIDATION PROCESS:
1. Cross-check edges:
- Verify each edge using synthesis consensus
- Remove edges with low confidence (< 5/7 agreement)
- Add missing edges identified by consensus
2. Identify hidden cascades:
- Indirect cascades (A → B → C, but A → C not obvious)
- Time-based cascades (A fails first, causes B later)
- State-based cascades (A leaves bad state, B fails on it)
3. Validate root cause claims:
For each claimed root:
- Verify no hidden dependencies
- Check if truly primary or just first detected
- Use 5-Whys: "Why did this fail?" → repeat 5 times
OUTPUT:
{
validatedGraph: { /* corrected graph */ },
validatedRoots: [ /* confirmed roots */ ],
hiddenCascades: [ /* newly discovered */ ],
conflicts: [ /* disagreements with Analyst 1 */ ]
}
Store: .claude/.artifacts/failure-graph-2.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "graph-2"`,
"analyst")
// Wait for graph analysis
npx claude-flow@alpha task wait --namespace "cicd/graph-analysis"
```
### Phase 2: Connascence Analysis
**Identify code coupling that affects fix strategy**
```javascript
[Single Message - Parallel Connascence Detection]:
Task("Connascence Detector (Name)",
`Scan for connascence of name: shared variable/function names causing failures.
Load root causes: .claude/.artifacts/failure-graph-2.json (validatedRoots)
For each root cause file:
1. Find all references to symbols (variables, functions, classes)
2. Identify which files import/use these symbols
3. Determine if failure requires changes across multiple files
Connascence of Name = When changing a name requires changing it everywhere
Impact on fixes:
- High connascence = Must fix all references atomically
- Low connascence = Can fix in isolation
Store: .claude/.artifacts/connascence-name.json
Use hooks: npx claude-flow@alpha hooks post-task --task-id "conn-name"`,
"code-analyzer")
Task("Connascence Detector (Type)",
`Scan for connascence of type: type dependencies causing failures.
Load root causes: .claude/.artifacts/failure-graph-2.json
For each root cause:
1. Identify type signature