workflow-optimizer
Skill from AIBPM42/hodgesfooshee-site-spark
- Category: discovered
- Tags: github, community
- Version: 1.0.0
- Author: walid
- Downloads: 0
- Updated: 2025-11-22
Download the latest version (v1.0.0) from this page.
SKILL.md
# Workflow Optimizer
## Description
Analyzes existing workflows and automation systems to identify bottlenecks, redundancies, and optimization opportunities. Specializes in N8N workflows, data pipelines, and multi-step automation processes. Use when you need to improve performance, reduce costs, or scale operations.
## When to Use This Skill
Trigger this skill when the user:
- Says workflow is "slow" or "expensive"
- Wants to "optimize" or "improve" existing automation
- Asks "how can we make this faster/cheaper?"
- Reports bottlenecks or failures in pipelines
- Needs to scale from X to 10X volume
- Wants to reduce manual intervention
- Asks about parallel processing or batching
## Core Capabilities
### 1. Bottleneck Identification
- Analyze workflow execution times
- Identify slowest steps
- Find sequential operations that could be parallel
- Detect rate limiting issues
- Spot unnecessary data transformations
### 2. Cost Optimization
- Calculate cost per operation
- Identify expensive API calls
- Recommend caching strategies
- Suggest batch processing
- Find redundant operations
### 3. Reliability Improvement
- Add error handling
- Implement retry logic
- Create fallback strategies
- Design circuit breakers
- Add monitoring and alerts
### 4. Scale Planning
- Identify scale bottlenecks
- Design for 10x/100x growth
- Plan infrastructure needs
- Optimize for throughput
- Balance cost vs speed
## Analysis Framework
### Step 1: Map Current State
```markdown
## Workflow: [Name]
### Flow Steps
1. [Step 1] - Duration: X, Cost: $Y
2. [Step 2] - Duration: X, Cost: $Y
3. [Step 3] - Duration: X, Cost: $Y
### Metrics
- Total execution time: [X seconds]
- Success rate: [Y%]
- Cost per run: $[Z]
- Runs per day: [N]
- Daily cost: $[N × Z]
### Pain Points
- [ ] Slow steps
- [ ] High failure rate
- [ ] Manual intervention needed
- [ ] High cost
- [ ] Can't scale
```
### Step 2: Identify Optimizations
For each step, evaluate:
**Speed Optimizations:**
- Can this run in parallel?
- Can we batch operations?
- Can we cache results?
- Can we reduce data transferred?
- Can we use a faster API/tool?
**Cost Optimizations:**
- Is this API call necessary?
- Can we use a cheaper alternative?
- Can we reduce call frequency?
- Can we negotiate volume pricing?
- Can we self-host?
**Reliability Optimizations:**
- What if this step fails?
- Do we have retry logic?
- Is there a fallback?
- Are we monitoring this?
- Can we recover automatically?
### Step 3: Prioritize Improvements
Use **Impact × Ease** matrix:
| Improvement | Impact | Ease | Priority |
|-------------|--------|------|----------|
| Run steps in parallel | High | Medium | 1 |
| Add caching | Medium | High | 2 |
| Batch API calls | High | Low | 3 |
**Priority Formula**: Impact (1-10) × Ease (1-10) = Score
- Score >50: Do immediately
- Score 25-50: Plan for this month
- Score <25: Nice to have
## Common Optimization Patterns
### Pattern 1: Parallel Processing
**Before:**
```
Step 1 → Step 2 → Step 3 → Step 4 → Step 5
Total: 50 seconds
```
**After:**
```
Step 1 → [Step 2, Step 3, Step 4 in parallel] → Step 5
Total: 20 seconds
```
**When to Use:**
- Steps don't depend on each other
- API allows concurrent requests
- System can handle parallel load
### Pattern 2: Batch Operations
**Before:**
```
For each item (1000 items):
- API call (100ms)
Total: 100 seconds
```
**After:**
```
Batch items into groups of 100:
- Batch API call (500ms)
Total: 5 seconds
```
**When to Use:**
- API supports batch requests
- Many small operations
- Rate limits allow batching
### Pattern 3: Smart Caching
**Before:**
```
Every request:
- Fetch data from API (1s)
- Transform data (0.1s)
Total: 1.1s per request
```
**After:**
```
First request:
- Fetch + cache (1s)
- Transform (0.1s)
Subsequent requests:
- Read from cache (0.01s)
- Transform (0.1s)
Total: 0.11s per request
```
**When to Use:**
- Data changes infrequently
- Same data accessed multiple times
- Cache invalidation is manageable
### Pattern 4: Smart Retries
**Before:**
```
API call fails → Workflow fails
Success rate: 85%
```
**After:**
```
API call fails → Wait 1s → Retry (3x max)
Success rate: 99%
```
**When to Use:**
- Transient failures are common
- Retry doesn't cause problems
- Cost of retry is acceptable
### Pattern 5: Staged Processing
**Before:**
```
Process all 10,000 items → 2 hours
(Can't stop, can't resume)
```
**After:**
```
Process in batches of 100:
- Batch 1 → Store progress
- Batch 2 → Store progress
- ...
(Can stop/resume anytime)
```
**When to Use:**
- Long-running operations
- Risk of failure mid-process
- Need progress tracking
## N8N-Specific Optimizations
### 1. Use Function Nodes for Simple Transforms
**Why**: Faster than HTTP nodes for basic operations
**Before**: HTTP node to external API for simple math
**After**: Function node with JavaScript
### 2. Minimize Data Between Nodes
**Why**: Large payloads slow execution
**Before**: Pass entire 10MB dataset
**After**: Pass only IDs, fetch details when needed
### 3. Use Split In Batches Node
**Why**: Process large datasets efficiently
```
[Split In Batches: 100 items]
↓
[Process Batch]
↓
[Merge Results]
```
### 4. Set Appropriate Timeouts
**Why**: Don't wait forever for failing operations
- Fast APIs: 5-10s timeout
- Slow APIs: 30-60s timeout
- Scraping: 60-120s timeout
### 5. Use Execute Once Per Item Wisely
**Why**: Can create unnecessary loops
**Slow**: Execute once per item for 1000 items = 1000 executions
**Fast**: Process array in single function = 1 execution
## Workflow Design Principles
### 1. Fail Fast
Validate inputs early, fail immediately on bad data.
```
[Validate Input] → [Expensive Operation]
Not: [Expensive Operation] → [Discover Bad Input] → [Fail]
```
### 2. Idempotent Operations
Design steps that can be retried safely.
**Good**: "Set user status to X" (same result if run twice)
**Bad**: "Increment counter by 1" (different result if run twice)
### 3. Observable Systems
Add logging and metrics at key points.
```
[Step 1] → [Log: "Step 1 complete"]
→ [Metric: duration_step1]
→ [Step 2]
```
### 4. Graceful Degradation
Design fallbacks for non-critical steps.
```
[Try: Enrich with API data]
↓ (If fails)
[Fallback: Use basic data]
↓ (Continue)
[Next Step]
```
## Cost Reduction Checklist
For each workflow, check:
- [ ] Are we caching API results?
- [ ] Are we batching requests?
- [ ] Are we using the cheapest API tier?
- [ ] Do we have unnecessary transformations?
- [ ] Are we filtering data early?
- [ ] Are we using free alternatives?
- [ ] Do we have redundant API calls?
- [ ] Are we processing duplicates?
- [ ] Can we reduce polling frequency?
- [ ] Are we storing unnecessary data?
## Performance Optimization Checklist
For each workflow, check:
- [ ] Can steps run in parallel?
- [ ] Are we using the fastest APIs?
- [ ] Do we have unnecessary waits?
- [ ] Are we transferring minimal data?
- [ ] Are we using efficient data formats?
- [ ] Do we have database indexes?
- [ ] Are we using connection pooling?
- [ ] Can we pre-compute results?
- [ ] Are we streaming large datasets?
- [ ] Do we have proper caching?
## Example Optimization Report
```markdown
## Optimization Report: Lead Hunter Pipeline
### Current Performance
- Execution time: 45 seconds per lead
- Cost: $0.50 per lead
- Success rate: 85%
- Throughput: 100 leads/hour
### Bottlenecks Identified
1. **Sequential skip tracing** (30s)
- FastPeopleSearch: 15s
- TruePeopleSearch: 15s
- Running sequentially
2. **No caching** (Cost: +$0.20/lead)
- Re-validating known emails
- Re-checking business patterns
3. **No retry logic** (Success: 85%)
- API failures not retried
- Lost 15% of leads
### Recommended Optimizations
#### Quick Wins (Implement Today)
1. **Parallel Skip Tracing**
- Impact: -20s execution time
- Effort: 1 hour
- Savings: 44% time reduction
2. **Add Validation Cache**
- Impact: -$0.20 per lead
- Effort: 2 hours
- Savings: $200/month (1000 leads)
3. **Retry Failed API Calls**
- Impact: 85% → 98% success
- Effort: 1 hour
- Gains: +15% more leads
#### 30-Day Improvements
4. **Batch Property Lookups**
- Impact: -10s execution time
- Effort: 1 day
- Savings: 22% time reduction
5. **Smart Lead Scoring**
- Impact: Process high-value first
- Effort: 2 days
- Business value: +30% conversion
#### 90-Day Vision
6. **Predictive Lead Filtering**
- Impact: Reduce low-value processing
- Effort: 1 week
- Savings: 40% cost reduction
### Projected Results
**After Quick Wins:**
- Execution time: 25s (-44%)
- Cost: $0.30 (-40%)
- Success rate: 98% (+15%)
- Throughput: 144 leads/hour (+44%)
**ROI**: 6 hours effort → $200/month savings = Break-even in 1 day
### Implementation Plan
**Week 1:**
- [ ] Parallel skip tracing (Day 1)
- [ ] Validation cache (Day 2-3)
- [ ] Retry logic (Day 4)
- [ ] Test and deploy (Day 5)
**Weeks 2-4:**
- [ ] Batch property lookups
- [ ] Smart lead scoring
**Month 2-3:**
- [ ] Predictive filtering
- [ ] A/B test optimization
```
## Integration with Other Skills
- **data-strategy-architect**: For high-level architecture decisions
- **error-annihilator**: When optimizations cause bugs
- **lead-hunter**: For domain-specific Lead Hunter optimizations
- **roadmap-builder**: To prioritize optimization work
## Key Metrics to Track
### Performance Metrics
- Execution time (p50, p95, p99)
- Throughput (ops/second, ops/hour)
- Latency per step
- Queue depth
### Reliability Metrics
- Success rate
- Error rate by type
- Retry frequency
- Recovery time
### Cost Metrics
- Cost per operation
- API calls per operation
- Data transfer costs
- Infrastructure costs
### Business Metrics
- Cost per lead
- Time to result
- Lead quality score
- Conversion rate
---
**Last Updated**: November 21, 2025
**Version**: 1.0