engineering-operations-management
Examples of my Claude Code infrastructure with skill auto-activation, hooks, and agents
- Category: discovered
- Tags: github, community
- Version: 1.0.0
- Author: walid
- Downloads: 0
- GitHub stars: 2
- Updated: 2025-11-21
Download the latest version (v1.0.0) from this page.
SKILL.md
# Engineering Operations Management Skill
**For managers running SRE, platform, and infrastructure teams - focusing on operations, on-call, incidents, and engineering metrics.**
> This skill helps engineering managers build sustainable operations practices, prevent burnout, run effective incident reviews, and measure what matters. Complements technical SRE skills with people and process management.
---
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🎯 SKILL ACTIVATED: engineering-operations-management
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## When to Use This Skill
**You're a manager who needs help with:**
- Structuring on-call rotations and compensation
- Preventing and addressing engineer burnout
- Running blameless postmortem reviews
- Negotiating SLOs with product teams
- Measuring engineering productivity (not vanity metrics)
- Managing toil and operational load
- Balancing feature work vs operational excellence
- Building sustainable operations culture
**This skill does NOT cover:**
- Hands-on incident response (see **sre** skill)
- Technical SLO/SLI implementation (see **sre** skill)
- Monitoring/observability setup (see **platform-engineering** skill)
- Team hiring and career development (see **engineering-management** skill)
---
## Core Areas
### 1. On-Call Management
**Core principle:** On-call is a necessary operational burden that should be **fair, sustainable, and compensated**.
#### On-Call Rotation Models
**Follow-the-Sun (Recommended for global teams):**
```
Americas Team: 6 AM - 2 PM PST (primary)
EMEA Team: 2 PM - 10 PM PST (primary)
APAC Team: 10 PM - 6 AM PST (primary)
Pros: No one wakes up at night, balanced load
Cons: Requires global team, handoff complexity
Best for: Teams with 15+ engineers across timezones
```
**Weekly Rotation (Most common):**
```
Week 1: Engineer A (primary), Engineer B (secondary)
Week 2: Engineer C (primary), Engineer D (secondary)
Week 3: Engineer E (primary), Engineer A (secondary)
Pros: Simple, fair rotation
Cons: Weekend coverage, potential burnout
Best for: Teams with 6-10 engineers in same timezone
```
**Tiered Escalation:**
```
Tier 1: Junior engineers (business hours only)
Tier 2: Senior engineers (24/7 primary)
Tier 3: Staff/Principal (escalation only)
Pros: Gradual responsibility increase
Cons: Can create "us vs them" dynamic
Best for: Large teams (15+) with clear skill levels
```
#### On-Call Compensation Models
**Option 1: On-Call Stipend**
```
Primary on-call: $500-$1,000/week
Secondary on-call: $250-$500/week
Pros: Predictable, simple
Cons: Doesn't account for actual pages
```
**Option 2: Pay-per-Page**
```
Business hours page: $50-$100
After-hours page: $150-$300
Weekend/holiday page: $300-$500
Pros: Fair - pays for actual disruption
Cons: Can incentivize ignoring issues
```
**Option 3: Hybrid (Recommended)**
```
Base stipend: $500/week
+ $100 per after-hours page
+ Comp time (1.5x hours worked after-hours)
Pros: Covers both availability and interruptions
Cons: More complex to administer
```
**Comp time policies:**
- For every hour worked after-hours, grant 1.5 hours comp time
- Comp time must be used within 30 days
- Encourage taking comp time day-after major incidents
#### Burnout Prevention
**Warning signs:**
- 🚨 Pages > 5 per week for extended period
- 🚨 Engineer mentions exhaustion, stress in 1-on-1s
- 🚨 Quality of work declining
- 🚨 Working late nights/weekends regularly
- 🚨 Cynicism, disengagement
**Interventions:**
1. **Immediate:** Rotate off on-call for 2-4 weeks
2. **Short-term:** Reduce project load, pair with senior engineer
3. **Long-term:** Fix underlying system issues causing pages
**Sustainable on-call rules:**
- No engineer on-call more than 1 week per month
- Maximum 2 weeks on-call per quarter
- Mandatory break after high-page-volume week
- No on-call during PTO or major life events
#### On-Call Scenarios
**Scenario: "What's a fair after-hours pay model?"**
- **Hybrid model (recommended):**
- Base: $500/week on-call stipend
- Plus: $100-150 per after-hours page
- Plus: 1.5x comp time for hours worked
- **Example calculation:**
- Week stipend: $500
- 3 after-hours pages × $125 = $375
- 4 hours worked × 1.5 = 6 hours comp time
- **Total value:** $875 + 6 hours off
**Scenario: "Team blamed someone in incident review - how to fix?"**
- **Immediate:** Stop the review, reset the tone
- **Say:** "We don't blame people, we fix systems. Let's focus on what failed, not who."
- **Blameless culture principles:**
- People make reasonable decisions based on information available
- Systems should prevent single points of failure
- Focus on "what" not "who"
- **Follow-up:** Coach manager running review on blameless principles
**Scenario: "During incident, what should I do as manager?"**
- **Monitor:** Watch incident channel, don't interrupt
- **Support:** "What do you need? More people? Communication handled?"
- **Shield:** Handle exec questions, keep pressure off team
- **Don't:** Take over, second-guess, or ask "why" questions mid-incident
- **After:** Thank team, schedule postmortem, ensure comp time taken
**Scenario: "How do we track incident trends?"**
- **Metrics to track:**
- Incident frequency (per week/month)
- MTTR (mean time to recovery)
- Incidents by service/component
- Incidents by root cause category
- **Look for patterns:**
- Same service failing repeatedly → systemic issue
- MTTR increasing → lack of familiarity or tooling gaps
- Spike in incidents → recent deploy or infrastructure change
- **Action:** Address top 3 incident sources quarterly
**Scenario: "What incident communication plan do we need?"**
- **During incident:**
- Sev 1: Updates every 30 minutes to execs, status page every 15 min
- Sev 2: Updates every hour to stakeholders
- Sev 3: Update when resolved
- **Channels:**
- Internal: Dedicated Slack #incidents channel
- External: Status page (Statuspage.io, etc.)
- Executives: Email + Slack DM for Sev 1/2
- **Template:**
```
[SEV 1] API Service Outage
Impact: All users unable to login
Status: Investigating
Next update: 2:30 PM (15 minutes)
```
**Scenario: "What's the right retrospective format?"**
- **Timeline:** Within 48 hours of incident (while fresh)
- **Attendees:** Incident responders + anyone interested (open invitation)
- **Duration:** 45-60 minutes
- **Format:**
1. Timeline walkthrough (10 min)
2. What went well (10 min)
3. What went poorly (15 min)
4. Action items (15 min) - with owners and due dates
5. Q&A (10 min)
- **Output:** Written postmortem + action items tracked
**Scenario: "How do we communicate incidents to executives?"**
- **During:** Brief, factual updates
- "API down, 100% of users affected, team investigating"
- **After:** Business-focused summary
- Revenue impact: "$50K in lost sales"
- User impact: "10K users couldn't check out for 2 hours"
- Prevention: "Adding rate limiting to prevent recurrence"
- **Avoid:** Deep technical details unless asked
---
### 2. Incident Management for Managers
**Your role as a manager during incidents:**
#### During the Incident (DO NOT take over unless critical)
```
✅ DO:
- Monitor incident channel, offer support
- Shield team from external pressure
- Bring in additional engineers if needed
- Coordinate with stakeholders (updates to execs)
- Order food if it's going long
- Take notes for postmortem
❌ DON'T:
- Take over incident response (unless you're most qualified)
- Ask "why didn't you..." questions during incident
- Pressure for faster resolution
- Blame individuals
- Second-guess decisions being made
```
**Incident Severity Levels (align with team):**
```
Sev 1 (Critical):
├── Complete service outage
├── Data loss or security breach
├── Revenue impact > $10K/hour
└── Response: All hands, exec updates every 30 min
Sev 2 (High):
├── Major feature degraded
├── Significant user impact
├── Revenue impact > $1K/hour
└── Response: On-call + expert, updates every hour
Sev 3 (Medium):
├── Minor feature degraded
├── Limited user impact
└── Response: On-call handles, regular updates
Sev 4 (Low):
├── Internal tooling issue
├── No user impact
└── Response: Fix during business hours
```
#### After the Incident: Blameless Postmortem
**Blameless postmortem framework:**
```
Postmortem Template:
## Incident Summary
- Date/Time: When did it happen?
- Duration: How long?
- Impact: Who was affected? How many users?
- Severity: Sev 1-4
## Timeline
- 14:32 - First alert fired
- 14:35 - Engineer A acknowledged, began investigation
- 14:45 - Root cause identified (database connection pool exhausted)
- 15:00 - Mitigation applied (increased pool size)
- 15:15 - Service fully recovered
## Root Cause
What actually caused this? (Technical, not "Engineer X did...")
## What Went Well
- Alert fired within 2 minutes
- Communication was clear
- Rollback was smooth
## What Went Poorly
- No automated mitigation
- Monitoring didn't catch early warning signs
- On-call engineer not familiar with this service
## Action Items
1. [P0] Add automated connection pool scaling (Owner: Alice, Due: 2 weeks)
2. [P1] Improve monitoring for connection pool saturation (Owner: Bob, Due: 1 month)
3. [P2] Add service to on-call training rotation (Owner: Manager, Due: 2 weeks)
## Lessons Learned
- Database connection pool defaults are too conservative
- Need better pre-production load testing
```
**Blameless postmortem meeting (45-60 min):**
```
1. Introduction (5 min)
└── Remind: This is blameless, focus on systems not people
2. Timeline Review (15 min)
└── Walk through what happened, when
3. Root Cause Analysis (15 min)
└── "Why did this happen?" (ask "why" 5 times)
4. What Went Well / What Went Poorly (10 min)
└── Balanced reflection
5. Action Items (10 min)
└── Specific, assigned, with due dates
└── Priority: P0 (this week), P1 (this month), P2 (nice to have)
6. Close (5 min)
└── Thank the team, emphasize learning
```
**Red flags in postmortems:**
- ❌ Blaming individuals ("Alice should have...")
- ❌ Vague action items ("Improve monitoring")
- ❌ No follow-up on action items
- ❌ Defensive posturing
- ❌ Skipping postmortems for "small" incidents
**Manager's job:** Enforce blameless culture, track action items, ensure learning.
---
### 3. SLO Negotiation with Product Teams
**The tension:** Product wants features fast. SRE/Platform wants stability. You balance both.
#### Understanding SLOs (Simple Version for Managers)
```
SLI (Service Level Indicator):
What you measure (e.g., "API latency p99")
SLO (Service Level Objective):
Target for reliability (e.g., "API latency p99 < 500ms, 99.9% of the time")
SLA (Service Level Agreement):
Contractual promise to customers (e.g., "99.95% uptime or we give refund")
Example:
SLI: Request success rate
SLO: 99.9% of requests succeed (internal target)
SLA: 99.5% uptime (customer-facing promise)
```
**Error budget concept:**
```
SLO: 99.9% availability = 0.1% allowed downtime
Per month (30 days):
├── Total time: 43,200 minutes
├── Allowed downtime: 43.2 minutes
└── Error budget: 43.2 minutes
If error budget exhausted:
├── Freeze feature releases
├── Focus on reliability improvements
└── Pay down tech debt
```
#### SLO Negotiation Framework
**When product pushes for aggressive feature timeline:**
```
Product: "We need to ship this feature in 2 weeks"
You (as manager):
"Let's check our error budget first. If we have budget, we can move fast.
If we're out of budget, we need to stabilize first."
Scenario 1: Error budget healthy (50% remaining)
├── ✅ Green light for feature work
├── 70% capacity on features
└── 30% on reliability
Scenario 2: Error budget exhausted (0% remaining)
├── 🛑 Feature freeze
├── 100% capacity on reliability
└── Resume features when budget recovers
```
**How to set SLOs (practical guide):**
1. **Start with current performance:**
- "Our API latency p99 is currently 300ms"
- Don't set SLO at 300ms - give yourself buffer
2. **Set realistic target:**
- "Let's set SLO at p99 < 500ms"
- This gives 200ms buffer for growth/issues
3. **Align with customer expectation:**
- "Customers complain if latency > 1s"
- SLO should prevent customer pain
4. **Review quarterly:**
- Too easy? (Always meeting SLO) → Tighten SLO or invest in features
- Too hard? (Always missing SLO) → Loosen SLO or invest in reliability
**Common SLOs by service type:**
```
API Services:
├── Availability: 99.9% (43 min downtime/month)
├── Latency p50: < 100ms
├── Latency p99: < 500ms
└── Error rate: < 0.1%
Batch Processing:
├── Job success rate: 99.5%
├── Job completion time: < 4 hours
└── Data accuracy: 99.99%
Data Pipeline:
├── Data freshness: < 15 min lag
├── Pipeline availability: 99.9%
└── Data quality: 99.95%
```
---
### 4. Engineering Metrics That Matter
**The problem:** Easy to measure vanity metrics. Hard to measure real productivity.
#### Vanity Metrics (Avoid)
```
❌ Lines of code written
❌ Number of commits
❌ Hours worked
❌ Number of deploys (without context)
❌ Ticket velocity (without quality)
❌ Code coverage % (without context)
```
**Why these are bad:**
- Lines of code: Good engineers often delete code
- Number of commits: Encourages small, meaningless commits
- Hours worked: Encourages burnout, not productivity
- Deploys without context: Could be hotfixes for bugs you introduced
- Ticket velocity: Encourages cherry-picking easy tickets
- Code coverage: Can write useless tests to hit %
#### Metrics That Actually Matter
**1. DORA Metrics (Use these)**
```
Deployment Frequency:
├── How often do you deploy to production?
├── Elite: Multiple times per day
├── High: Daily to weekly
├── Medium: Weekly to monthly
└── Low: Monthly to every 6 months
Lead Time for Changes:
├── How long from commit to production?
├── Elite: < 1 hour
├── High: 1 day to 1 week
├── Medium: 1 week to 1 month
└── Low: 1 month to 6 months
Time to Restore Service:
├── How long to recover from incident?
├── Elite: < 1 hour
├── High: < 1 day
├── Medium: 1 day to 1 week
└── Low: > 1 week
Change Failure Rate:
├── What % of changes cause incidents?
├── Elite: 0-15%
├── High: 16-30%
├── Medium: 31-45%
└── Low: > 45%
```
**How to use DORA metrics:**
- Track quarterly, not daily (avoid gaming)
- Trend over time (are we improving?)
- Compare to benchmarks (elite, high, medium, low)
- Use to identify improvement areas
**2. SRE Metrics**
```
Toil Percentage:
├── What % of engineer time is manual ops work?
├── Target: < 30% toil
├── Intervention needed: > 50% toil
└── Measure: Time tracking, surveys
On-Call Load:
├── Pages per week per engineer
├── Target: < 3 pages/week
├── Intervention: > 5 pages/week
└── Measure: PagerDuty analytics
SLO Compliance:
├── Are we meeting our SLOs?
├── Target: 99%+ SLO compliance
└── Measure: Observability dashboards
```
**3. Team Health Metrics**
```
Engineer Satisfaction:
├── Quarterly survey (1-10 scale)
├── Questions: "Satisfied with work?", "Would recommend team?"
├── Target: 8+ average
└── Red flag: < 6 average or declining trend
Retention Rate:
├── % of engineers staying > 1 year
├── Target: > 85% annual retention
└── Red flag: < 70% retention
Time to Productivity (New hires):
├── How long until new hire is productive?
├── Target: < 90 days
└── Measure: Manager assessment + self-assessment
```
**4. Operational Excellence Metrics**
```
Incident Trends:
├── Number of Sev 1/2 incidents per month
├── Target: Declining or stable
└── Red flag: Increasing trend
Postmortem Action Item Completion:
├── % of action items completed on time
├── Target: > 80% completion
└── Red flag: < 50% completion
Automated Test Coverage:
├── % of critical paths covered
├── Target: > 70% for critical paths
└── Not a vanity metric if focused on high-risk areas
```
#### How to Present Metrics to Leadership
**Dashboard structure:**
```
1. Health at a Glance (Top metrics)
├── 🟢 SLO Compliance: 99.8% (Target: 99%)
├── 🟡 Deployment Frequency: 3x/week (Target: Daily)
├── 🟢 Incident Rate: 2 Sev2 this month (Last month: 4)
└── 🟢 Team Satisfaction: 8.2/10 (Target: 8+)
2. DORA Metrics Trend (Quarterly)
[Chart showing improvement over time]
3. Focus Areas
├── ✅ Reduced incident rate by 50% this quarter
├── 🚧 Working on deployment frequency (automation initiative)
└── ⚠️ Toil still high at 40% - hiring 2 more engineers
4. Asks
├── Budget for observability tooling ($50K)
└── Approval to pause feature work next sprint for reliability
```
---
### 5. Balancing Feature Work vs Operational Excellence
**The eternal tension:** Product wants features. You want stability.
#### Resource Allocation Models
**70-20-10 Rule (Recommended):**
```
70% Feature Work:
├── New features product wants
├── Customer-facing improvements
└── Revenue-generating projects
20% Operational Excellence:
├── Tech debt paydown
├── Reliability improvements
├── Monitoring enhancements
└── Automation
10% Innovation/Learning:
├── Explore new technologies
├── Hackathons
├── Learning time
└── Experimentation
```
**Adjust based on phase:**
```
High Growth Phase:
├── 80% Features
├── 15% Ops Excellence
└── 5% Innovation
Stability Phase:
├── 50% Features
├── 40% Ops Excellence
└── 10% Innovation
Crisis Phase (Post-Incidents):
├── 30% Features
├── 60% Ops Excellence
└── 10% Innovation
```
#### Negotiating with Product
**When product asks for all-feature, no-ops time:**
```
Scenario: "We need all engineers on Feature X for Q4"
Your response framework:
1. Acknowledge business need
"I understand Feature X is critical for revenue"
2. State operational reality
"Our on-call load is high (8 pages/week) and error budget is 80% exhausted"
3. Present options
Option A: All-in on features, risk of incidents and burnout
Option B: 70-30 split, sustainable pace, less feature risk
Option C: Hire 2 more engineers to do both
4. Recommend
"I recommend Option B - we'll deliver 70% of Feature X this quarter,
and ensure we don't have outages that impact customers"
5. Make it their decision
"What's your preference given these trade-offs?"
```
**Using error budgets as negotiation tool:**
```
Error budget = objective metric, not subjective
If product wants to move fast:
├── Check error budget: 50% remaining?
├── ✅ Green light: "We have budget, let's ship!"
└── 🛑 Budget exhausted: "We need to stabilize first"
This removes emotion from discussion. It's data-driven.
```
---
### 6. Building Sustainable Operations Culture
**Culture eats process for breakfast.**
#### Key Cultural Values
**1. Blameless Culture**
```
When incidents happen:
❌ "Who broke it?" → ✅ "What broke?"
❌ "Why didn't you..." → ✅ "What can we learn?"
❌ Hide mistakes → ✅ Share failures openly
```
**2. Automate Toil**
```
Manual work is not a badge of honor.
├── Track toil percentage
├── Reward automation, not heroics
└── "If you do it twice, automate it"
```
**3. Sustainable On-Call**
```
On-call is not punishment.
├── Fair rotation
├── Compensated fairly
├── Protected from burnout
└── Escalation is encouraged, not weakness
```
**4. Continuous Improvement**
```
Every incident is a learning opportunity.
├── Postmortems are required, not optional
├── Action items are tracked and completed
└── Celebrate fixes, not just features
```
#### Manager Actions to Reinforce Culture
**1. Lead by Example**
- Participate in on-call rotation (if you're technical)
- Admit your own mistakes publicly
- Take postmortem action items yourself
**2. Celebrate Operational Wins**
- Shout out engineers who reduce toil
- Highlight reliability improvements in team meetings
- Give "Operational Excellence" awards
**3. Protect Your Team**
- Say no to unrealistic timelines
- Push back on "just ship it" pressure
- Shield team from org politics
**4. Invest in Automation**
- Allocate 20% capacity to ops excellence
- Approve tool/platform budgets
- Hi