dss/docs/TOKEN_INGESTION.md

# DSS Token Ingestion Guide

Complete documentation for multi-source token ingestion, merge strategies, and translation dictionaries.

## Table of Contents

1. [Overview](#overview)
2. [Supported Sources](#supported-sources)
3. [Token Extraction](#token-extraction)
4. [Merge System](#merge-system)
5. [Translation Dictionaries](#translation-dictionaries)
6. [MCP Tools](#mcp-tools)
7. [Examples](#examples)

---

## Overview

DSS provides a unified token ingestion pipeline that:

- Extracts tokens from multiple source formats (CSS, SCSS, Tailwind, JSON, Figma)
- Normalizes naming conventions to DSS canonical format
- Merges tokens with intelligent conflict resolution
- Generates translation dictionaries for traceability
- Exports to multiple output formats

```
┌─────────────────────────────────────────────────────────────────────┐
│                     DSS INGESTION PIPELINE                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   SOURCES              PROCESS                OUTPUT                 │
│   ───────              ───────                ──────                 │
│   CSS        ─┐                                                      │
│   SCSS       ─┼─→  Extract  →  Normalize  →  Merge  →  Export       │
│   Tailwind   ─┤       ↓           ↓           ↓          ↓          │
│   JSON/W3C   ─┤    Tokens    DSS Names    Unified    CSS/SCSS/      │
│   Figma      ─┘                                      TS/JSON         │
│                                                                      │
│                            ↓                                         │
│                   Translation Dictionary                             │
│                   (per-project mapping)                              │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Supported Sources

### CSS Custom Properties

Extracts CSS variables from `:root` and other selectors.

```css
:root {
  /* Primary colors */
  --primary-500: #3B82F6;
  --primary-600: #2563EB;

  /* Spacing */
  --spacing-md: 16px;
}
```

**Features:**
- Comment extraction for descriptions
- Auto-detection of token type (color, dimension, etc.)
- Category inference from naming patterns

### SCSS Variables

Extracts SCSS variables and maps.

```scss
// Brand colors
$primary-500: #3B82F6;
$primary-600: #2563EB;

// Spacing map
$spacing: (
  xs: 4px,
  sm: 8px,
  md: 16px,
  lg: 24px,
);
```

**Features:**
- Single variable extraction
- Map/object flattening
- Comment-based descriptions

### Tailwind Configuration

Extracts tokens from `tailwind.config.js/ts` or Tailwind v4 CSS.

```javascript
// tailwind.config.js
module.exports = {
  theme: {
    extend: {
      colors: {
        primary: {
          500: '#3B82F6',
          600: '#2563EB',
        },
      },
    },
  },
};
```

**Features:**
- Theme object parsing
- Extend section support
- Tailwind v4 `@theme` directive

### JSON Token Files

Supports multiple JSON formats:

**W3C Design Tokens:**
```json
{
  "color": {
    "primary": {
      "500": {
        "$value": "#3B82F6",
        "$type": "color",
        "$description": "Primary brand color"
      }
    }
  }
}
```

**Style Dictionary:**
```json
{
  "color": {
    "primary": {
      "500": {
        "value": "#3B82F6",
        "comment": "Primary brand color"
      }
    }
  }
}
```

**Tokens Studio (Figma Plugin):**
```json
{
  "global": {
    "color": {
      "primary": {
        "value": "#3B82F6",
        "type": "color"
      }
    }
  }
}
```

---

## Token Extraction

### Basic Usage

```python
from tools.ingest import CSSTokenSource, SCSSTokenSource, JSONTokenSource

# Extract from CSS
css_source = CSSTokenSource()
css_tokens = await css_source.extract("/path/to/tokens.css")

# Extract from SCSS
scss_source = SCSSTokenSource()
scss_tokens = await scss_source.extract("/path/to/variables.scss")

# Extract from JSON
json_source = JSONTokenSource()
json_tokens = await json_source.extract("/path/to/tokens.json")
```

### Token Structure

Each extracted token contains:

```python
@dataclass
class DesignToken:
    name: str              # Normalized name: "color.primary.500"
    value: Any             # Token value: "#3B82F6"
    type: TokenType        # Type enum: COLOR, DIMENSION, etc.
    description: str       # From comments/descriptions
    source: str            # Source identifier: "css:tokens.css:12"
    source_file: str       # Original file path
    source_line: int       # Line number in source
    original_name: str     # Original name: "--primary-500"
    original_value: str    # Original value before processing
    category: TokenCategory # Category: COLORS, SPACING, etc.
    tags: List[str]        # Custom tags
    deprecated: bool       # Deprecation flag
    version: str           # Version string
```

### Name Normalization

All source names are normalized to DSS canonical format:

| Source Format | Original Name | DSS Canonical |
|---------------|---------------|---------------|
| CSS | `--primary-500` | `primary.500` |
| CSS (prefixed) | `--heroui-primary-500` | `heroui.primary.500` |
| SCSS | `$primary-500` | `primary.500` |
| camelCase | `--brandAccent` | `brand.accent` |
| SCREAMING | `--SPACING_LG` | `spacing.lg` |

---

## Merge System

### Merge Strategies

| Strategy | Description | Best For |
|----------|-------------|----------|
| `FIRST` | Keep first occurrence | Preserving original values |
| `LAST` | Use latest value (override) | Latest source wins |
| `PREFER_FIGMA` | Prioritize Figma sources | Design-led workflow |
| `PREFER_CODE` | Prioritize CSS/SCSS sources | Code-led workflow |
| `PREFER_SPECIFIC` | Prefer concrete values over `var()` | Resolving references |
| `MERGE_METADATA` | Combine metadata, use latest value | Preserving history |
| `ERROR` | Raise error on conflict | Strict validation |
| `INTERACTIVE` | Require user decision | Manual review |

### Usage

```python
from tools.ingest import TokenMerger, MergeStrategy

# Create merger with strategy
merger = TokenMerger(strategy=MergeStrategy.PREFER_FIGMA)

# Merge multiple collections
result = merger.merge([css_tokens, scss_tokens, figma_tokens])

# Access results
print(f"Total tokens: {result.stats['total_tokens']}")
print(f"Conflicts resolved: {result.stats['conflicts_resolved']}")

# Examine conflicts
for conflict in result.conflicts:
    print(f"Token: {conflict.token_name}")
    print(f"  Source A: {conflict.existing.value} ({conflict.existing.source})")
    print(f"  Source B: {conflict.incoming.value} ({conflict.incoming.source})")
    print(f"  Resolution: {conflict.resolution}")
    print(f"  Final value: {conflict.resolved_token.value}")
```

### Custom Resolver

```python
def custom_resolver(conflict: MergeConflict) -> DesignToken:
    """Custom conflict resolution logic."""
    # Always prefer hex colors over named colors
    if conflict.incoming.value.startswith('#'):
        return conflict.incoming
    return conflict.existing

merger = TokenMerger(
    strategy=MergeStrategy.LAST,
    custom_resolver=custom_resolver
)
```

### Merge Result

```python
@dataclass
class MergeResult:
    collection: TokenCollection  # Merged tokens
    conflicts: List[MergeConflict]  # All conflicts encountered
    stats: Dict[str, int]  # Statistics
    warnings: List[str]  # Warning messages
```

Statistics include:
- `total_tokens`: Final token count
- `new_tokens`: Tokens added without conflict
- `updated_tokens`: Tokens updated during merge
- `conflicts_resolved`: Conflicts auto-resolved
- `conflicts_unresolved`: Conflicts requiring review

---

## Translation Dictionaries

### Purpose

Translation dictionaries map external token systems TO DSS canonical structure:

1. **Traceability** - Know where each token came from
2. **Reproducibility** - Re-run ingestion with same mappings
3. **Documentation** - Human-readable mapping reference
4. **Custom Props** - Track client-specific extensions

### Schema

```json
{
  "$schema": "dss-translation-v1",
  "project": "project-name",
  "source": "heroui | shadcn | css | scss | figma | tailwind",
  "version": "1.0.0",
  "created": "2025-01-15",
  "mappings": {
    "tokens": {
      "<source-token>": "<dss-canonical-token>"
    },
    "components": {
      "<source-component>": "<dss-component>[variant=value]"
    }
  },
  "custom_props": {
    "<dss-namespaced-token>": "<value>"
  },
  "unmapped": ["tokens that couldn't be mapped"],
  "notes": ["human-readable notes"]
}
```

### Project Structure

```
project-acme/
├── .dss/
│   ├── config.json           # Project configuration
│   └── translations/
│       ├── heroui.json       # HeroUI → DSS mappings
│       ├── legacy-css.json   # Legacy CSS → DSS mappings
│       └── custom.json       # Custom props specific to ACME
```

---

## MCP Tools

DSS provides MCP tools for token ingestion:

### `ingest_css_tokens`
Extract tokens from CSS file.
```
ingest_css_tokens(source: "/path/to/tokens.css")
```

### `ingest_scss_tokens`
Extract tokens from SCSS file.
```
ingest_scss_tokens(source: "/path/to/variables.scss")
```

### `ingest_tailwind_tokens`
Extract tokens from Tailwind config.
```
ingest_tailwind_tokens(source: "/path/to/tailwind.config.js")
```

### `ingest_json_tokens`
Extract tokens from JSON file (W3C, Style Dictionary, Tokens Studio).
```
ingest_json_tokens(source: "/path/to/tokens.json")
```

### `merge_tokens`
Merge multiple token sources.
```
merge_tokens(
  sources: "/path/a.css,/path/b.scss",
  strategy: "prefer_figma"
)
```

### `export_tokens`
Export tokens to various formats.
```
export_tokens(
  source: "/path/to/tokens.css",
  format: "typescript",  # css, scss, typescript, json, tailwind
  output_path: "/path/to/output.ts"
)
```

### `validate_tokens`
Validate token collection.
```
validate_tokens(source: "/path/to/tokens.css")
```

---

## Examples

### Example 1: Merge HeroUI + Legacy CSS

```python
import asyncio
from tools.ingest import (
    CSSTokenSource, TokenMerger, MergeStrategy
)

async def merge_heroui_legacy():
    css_source = CSSTokenSource()

    # Extract from both sources
    heroui_tokens = await css_source.extract("heroui-theme.css")
    legacy_tokens = await css_source.extract("legacy-styles.css")

    # Merge with HeroUI priority (newer system)
    merger = TokenMerger(strategy=MergeStrategy.LAST)
    result = merger.merge([legacy_tokens, heroui_tokens])

    print(f"Merged {result.stats['total_tokens']} tokens")
    print(f"Resolved {len(result.conflicts)} conflicts")

    # Export to TypeScript
    print(result.collection.to_typescript())

    return result

asyncio.run(merge_heroui_legacy())
```

### Example 2: Build Translation Dictionary

```python
import json
from tools.ingest import CSSTokenSource

async def build_translation():
    css_source = CSSTokenSource()
    tokens = await css_source.extract("heroui-theme.css")

    translation = {
        "$schema": "dss-translation-v1",
        "project": "heroui-migration",
        "source": "heroui",
        "mappings": {
            "tokens": {}
        }
    }

    # Build mappings
    for token in tokens.tokens:
        # Map HeroUI naming to DSS canonical
        dss_name = token.normalize_name()

        # Convert: heroui.primary.500 → color.primary.500
        if "primary" in dss_name or "secondary" in dss_name:
            dss_name = f"color.{dss_name.replace('heroui.', '')}"

        translation["mappings"]["tokens"][token.original_name] = dss_name

    # Save translation dictionary
    with open(".dss/translations/heroui.json", "w") as f:
        json.dump(translation, f, indent=2)

    return translation
```

### Example 3: Diff Two Token Collections

```python
from tools.ingest import CSSTokenSource
from tools.ingest.merge import TokenDiff

async def compare_versions():
    css_source = CSSTokenSource()

    old_tokens = await css_source.extract("tokens-v1.css")
    new_tokens = await css_source.extract("tokens-v2.css")

    diff = TokenDiff.diff(old_tokens, new_tokens)

    print(TokenDiff.summary(diff))
    # Output:
    # Token Diff Summary:
    # ========================================
    #
    # + Added (5):
    #   + color.accent.500: #F59E0B
    #   + color.accent.600: #D97706
    #   ...
    #
    # - Removed (2):
    #   - color.deprecated.old: #FF0000
    #   ...
    #
    # ~ Changed (3):
    #   ~ color.primary.500: #3B82F6 → #2563EB
    #   ...
    #
    #   Unchanged: 120

asyncio.run(compare_versions())
```

---

## Best Practices

### 1. Always Use Translation Dictionaries

Even for simple projects, maintain translation dictionaries for:
- Audit trail of token origins
- Reproducible migrations
- Team documentation

### 2. Choose Appropriate Merge Strategy

| Scenario | Recommended Strategy |
|----------|---------------------|
| Design system update | `LAST` |
| Design-led project | `PREFER_FIGMA` |
| Code-led project | `PREFER_CODE` |
| Strict validation | `ERROR` |
| Complex migration | `MERGE_METADATA` |

### 3. Isolate Custom Props

Never pollute DSS core with client-specific tokens:

```json
{
  "custom_props": {
    "color.brand.acme.primary": "#1E40AF",
    "color.brand.acme.accent": "#F59E0B"
  }
}
```

### 4. Validate Before Production

Always run validation before deploying:

```python
result = await validate_tokens("/path/to/tokens.css")
if result["issues"]:
    print("Validation failed!")
    for issue in result["issues"]:
        print(f"  - {issue}")
```

### 5. Version Translation Dictionaries

Include version in translation dictionaries and track changes:

```json
{
  "version": "1.2.0",
  "created": "2025-01-15",
  "updated": "2025-01-20",
  "changelog": [
    "1.2.0: Added accent color mappings",
    "1.1.0: Updated primary scale",
    "1.0.0: Initial translation"
  ]
}
```