pipeline-runner

Skill from fstr21/sportsbot
Category: discovered
Tags: github, community
Version: 1.0.0
Author: walid
Downloads: 0
Updated: 2025-11-22
Source: fstr21/sportsbot
Download the latest version (v1.0.0) from this page.
SKILL.md

# Pipeline Runner Skill

**Purpose**: Run entire sport pipelines (CAPTURE → DERIVE → PACKAGE → INGEST) with one command.

## When Claude Should Use This Skill

Use this skill when:
- User says "run the NFL pipeline" or "run pipeline for NHL"
- User asks "capture today's games"
- User says "run full pipeline for [sport]"
- User asks "process [sport] data"
- User wants to run end-to-end pipeline for a date
- User says "get me tomorrow's games"

## What This Skill Does

1. **Orchestrates** full 4-stage pipeline for a sport
2. **Prompts** for date if not specified (defaults to today)
3. **Runs** CAPTURE stage (01-03 scripts)
4. **Runs** DERIVE stage (derived pipeline)
5. **Runs** PACKAGE stage (09-13 scripts) if available
6. **Runs** INGEST stage (database ingestor) if available
7. **Shows** progress at each stage (which script is running)
8. **Stops** if any stage fails (doesn't continue with bad data)
9. **Shows** summary of what was captured/derived/packaged/ingested
10. **Explains** results in plain English

## Critical Constraints

**MUST follow these rules**:
- NO emoji (Windows environment - use ASCII like [OK], [FAIL], [WARN])
- Run scripts in correct order (CAPTURE before DERIVE, etc.)
- Stop pipeline if critical failure occurs
- Display progress as pipeline runs (don't wait until end)
- Use subprocess to run Python scripts (not imports)
- Handle missing scripts gracefully (e.g., NBA has no packaging yet)
- Capture stdout/stderr from each script
- Set PYTHONPATH before running scripts (required for imports)

## How to Use This Skill

**Example 1**: User says "run the NFL pipeline"
- Claude runs: `python .claude/skills/pipeline-runner/scripts/run_pipeline.py --sport nfl`
- Script prompts: "Enter date (MM/DD/YYYY) or press Enter for today:"
- Claude shows: Progress through all 4 stages
- Claude explains: "Pipeline complete - 14 games captured and ingested"

**Example 2**: User says "capture tomorrow's NBA games"
- Claude runs: `python .claude/skills/pipeline-runner/scripts/run_pipeline.py --sport nba --date tomorrow`
- Claude shows: Progress through CAPTURE + DERIVE (stops before PACKAGE since NBA doesn't have it)
- Claude explains: "Captured 12 NBA games for 2025-10-24. Packaging not yet implemented."

**Example 3**: User says "run NHL pipeline for 2025-10-25"
- Claude runs: `python .claude/skills/pipeline-runner/scripts/run_pipeline.py --sport nhl --date 2025-10-25`
- Claude shows: Full pipeline execution
- Claude explains: "NHL pipeline complete - 3 games captured, derived, packaged, and ingested"

## Expected Outcome

After using this skill:
- User has run full pipeline with ONE command (not 10+ commands)
- User knows what was captured (game counts)
- User knows if any stage failed
- User knows data is in database (if ingest succeeded)
- User understands pipeline status (in plain English)
- User saved 10+ minutes of manual command execution

## Technical Details

The pipeline-runner script:
- `run_pipeline.py` - Main pipeline orchestrator

Pipeline flow:
1. **CAPTURE** (required):
   - `01_fetch_{sport}_odds.py`
   - `02_fetch_{sport}_stats.py`
   - `03_fetch_{sport}_public_betting.py`
2. **DERIVE** (required):
   - `pipelines/{sport}/derived/run_derived_pipeline.py`
3. **PACKAGE** (optional - may not exist):
   - `09_build_team_context.py`
   - `10_build_player_performance.py`
   - `11_build_matchup_summaries.py`
   - `12_build_manifest.py`
   - `13_build_slate_package.py`
4. **INGEST** (optional - may not exist):
   - `pipelines/{sport}/db/ingestor.py` or `load_artifacts.py`

All scripts:
- Run via subprocess (captures output in real-time)
- Set PYTHONPATH to project root before running
- Pass date parameter to each script
- Check exit codes to detect failures
- Display stdout/stderr as scripts run
- Accumulate summary stats (games captured, errors, etc.)

## Success Criteria

Script is successful if:
- Runs pipeline stages in correct order
- Stops if critical failure occurs
- Displays progress in real-time
- Shows summary of results
- Handles missing scripts gracefully (e.g., NBA has no packaging)
- Explains results in plain English
- User doesn't need to run 10+ commands manually

## Example Output

```
[OK] Starting NFL pipeline for 2025-10-23

==== STAGE 1: CAPTURE (3 scripts) ====

[OK] Running 01_fetch_nfl_odds.py...
Captured 14 games for 2025-10-23
[OK] Complete (32s)

[OK] Running 02_fetch_nfl_stats.py...
Captured 32 team stats
[OK] Complete (18s)

[OK] Running 03_fetch_nfl_public_betting.py...
Captured public betting for 14 games
[OK] Complete (45s)

==== STAGE 2: DERIVE (1 script) ====

[OK] Running run_derived_pipeline.py...
Processed 280 players (L5 averages, usage metrics)
[OK] Complete (12s)

==== STAGE 3: PACKAGE (5 scripts) ====

[OK] Running 09_build_team_context.py...
Built 32 team packages
[OK] Complete (5s)

... (scripts 10-13) ...

==== STAGE 4: INGEST (1 script) ====

[OK] Running ingestor.py...
Ingested 14 games, 280 players to database (0 errors)
[OK] Complete (8s)

==== PIPELINE COMPLETE ====

SUMMARY
[OK] Captured 14 NFL games
[OK] Derived metrics for 280 players
[OK] Packaged 32 team contexts + 280 player contexts
[OK] Ingested to database with 0 errors
[OK] Total time: 2m 15s

[OK] NFL pipeline complete for 2025-10-23
```