dvc-integration

Git-native documentation that grafts together as your sources evolve

Category: discovered
Tags: github, community
Version: 1.0.0
Author: walid
Downloads: 0
GitHub stars: 1
Updated: 2025-11-21

Source: danielnaab/graft

Download the latest version (v1.0.0) from this page.

SKILL.md

# Skill: DVC Integration

## Purpose
Help configure and maintain DVC (Data Version Control) pipelines for Graft artifacts, ensuring dvc.yaml stays synchronized with graft.yaml files.

## When to Use
Activate when the user mentions:
- DVC configuration
- dvc.yaml
- DVC pipelines
- Running dvc-scaffold
- DVC stages
- Pipeline setup

## DVC Overview for Graft

Graft uses DVC to:
- Track artifact dependencies and outputs
- Cache intermediate results
- Enable reproducible pipeline execution
- Share artifacts across team via remotes

The `graft dvc-scaffold` command generates `dvc.yaml` from `graft.yaml` files.

## Understanding the Integration

### Graft Artifact Structure
```yaml
# artifacts/my-artifact/graft.yaml
graft: my-artifact
inputs:
materials:
- path: ../source-data/data.csv
rev: main
derivations:
- id: processed
transformer: {ref: process}
template: {file: template.md, engine: jinja2}
outputs:
- path: output.md
```

### Generated DVC Stage
```yaml
# dvc.yaml (generated by graft dvc-scaffold)
stages:
my-artifact:
cmd: graft run artifacts/my-artifact/
deps:
- artifacts/my-artifact/graft.yaml
- artifacts/source-data/data.csv
outs:
- artifacts/my-artifact/output.md
```

## Using dvc-scaffold

### Generate DVC Pipeline
```bash
# From project root
graft dvc-scaffold .
```

This:
1. Finds all `graft.yaml` files in the project
2. Extracts dependencies (materials, templates, graft.yaml itself)
3. Extracts outputs from derivations
4. Generates stages mapping to `graft run` commands

### Verify Generated dvc.yaml
```bash
# Check the generated file
cat dvc.yaml

# Validate DVC can parse it
dvc status
```

### Run DVC Pipeline
```bash
# Run specific stage
dvc repro my-artifact

# Run entire pipeline
dvc repro

# Force re-run
dvc repro -f my-artifact
```

## Maintaining DVC Configuration

### When to Regenerate dvc.yaml

Regenerate after:
- Adding new artifacts
- Modifying inputs/materials in graft.yaml
- Changing output paths in graft.yaml
- Adding/removing derivations

**Process:**
1. Modify graft.yaml files
2. Run `graft dvc-scaffold .`
3. Review changes to dvc.yaml
4. Test with `dvc status`
5. Commit updated dvc.yaml

### Validating Synchronization

Check that dvc.yaml matches current graft.yaml files:

```bash
# Regenerate to temp file
graft dvc-scaffold . > /tmp/new-dvc.yaml

# Compare with existing
diff dvc.yaml /tmp/new-dvc.yaml
```

If differences exist and graft.yaml is correct, regenerate and commit.

## DVC Best Practices for Graft

### 1. Keep dvc.yaml in Git
```bash
git add dvc.yaml
git commit -m "chore: update DVC pipeline for new artifacts"
```

The `dvc.yaml` itself is tracked, but outputs are .dvc tracked.

### 2. Use DVC Remotes for Sharing
```bash
# Configure remote (one time)
dvc remote add -d storage s3://bucket/path
# Or local for testing:
dvc remote add -d storage /path/to/shared/cache

# Push artifacts to remote
dvc push

# Pull artifacts from remote
dvc pull
```

### 3. Commit .dvc Files
```bash
git add artifacts/**/output.md.dvc
git commit -m "feat: add processed artifacts"
```

### 4. Use .dvcignore Appropriately
Add to `.dvcignore`:
```
.venv/
__pycache__/
*.pyc
.pytest_cache/
```

## Common DVC Operations

### Check Pipeline Status
```bash
# See what needs to run
dvc status

# See what has changed
dvc diff
```

### Run Modified Stages
```bash
# Run only stages with changed dependencies
dvc repro

# Force specific stage
dvc repro -f my-artifact
```

### Visualize Pipeline
```bash
# Show pipeline DAG
dvc dag

# Show specific stage dependencies
dvc dag my-artifact
```

### Track New Outputs
If adding new outputs manually:
```bash
dvc add artifacts/my-artifact/new-output.csv
git add artifacts/my-artifact/new-output.csv.dvc .gitignore
```

## Troubleshooting

### Issue: dvc.yaml out of sync with graft.yaml
**Solution:**
```bash
graft dvc-scaffold .
git diff dvc.yaml # Review changes
git add dvc.yaml
git commit -m "chore: sync dvc.yaml with updated graft files"
```

### Issue: DVC can't find dependencies
**Symptom:** `ERROR: dependency 'path/to/file' does not exist`

**Solutions:**
- Ensure materialized files exist before running DVC
- Check paths in graft.yaml are correct and relative
- Verify materials are pulled/generated

### Issue: Stage always re-runs
**Causes:**
- Output files being modified outside DVC
- Timestamps changing
- Non-deterministic transformations

**Solutions:**
- Ensure `graft run` is truly deterministic
- Check nothing else is modifying outputs
- Use `dvc freeze <stage>` temporarily if needed

### Issue: Merge conflicts in dvc.yaml
**Solution:**
```bash
# Regenerate from current graft.yaml files
graft dvc-scaffold .
git add dvc.yaml
```

Since dvc.yaml is generated, regenerating is often simpler than manual merging.

## Integration with Graft Workflow

### Slice 4: DVC Scaffold Implementation
When implementing the dvc-scaffold command:
1. Parse all graft.yaml files in project
2. Extract materials → deps
3. Extract outputs → outs
4. Generate cmd: `graft run <artifact-path>`
5. Write stages to dvc.yaml

### After Slice 4: Using DVC
1. Create/modify graft.yaml
2. Run `graft dvc-scaffold .`
3. Review dvc.yaml changes
4. Run `dvc repro` to execute pipeline
5. Track outputs with DVC
6. Commit dvc.yaml and .dvc files

## Quality Checklist

- [ ] dvc.yaml generated successfully
- [ ] All graft artifacts have corresponding DVC stages
- [ ] Stage dependencies match graft.yaml materials
- [ ] Stage outputs match graft.yaml derivation outputs
- [ ] `dvc status` runs without errors
- [ ] `dvc dag` shows correct dependency graph
- [ ] Pipeline executes with `dvc repro`

## Advanced Patterns

### Conditional Stages
For artifacts with policies requiring human review:
```yaml
# In dvc.yaml, add manual approval step
stages:
my-artifact-draft:
cmd: graft run artifacts/my-artifact/
deps: [...]
outs: [artifacts/my-artifact/draft.md]

my-artifact-finalize:
cmd: graft finalize artifacts/my-artifact/ --agent human
deps: [artifacts/my-artifact/draft.md]
outs: [artifacts/my-artifact/final.md]
```

### Parallel Execution
DVC automatically parallelizes independent stages. Ensure graft artifacts are properly independent.

### Caching Strategy
```bash
# Configure cache location
dvc cache dir /fast/ssd/cache

# Share cache across projects
dvc config cache.shared group
```

## Outputs
- Synchronized dvc.yaml matching all graft artifacts
- Working DVC pipeline
- Proper dependency tracking
- Reproducible artifact generation