dss/docs/TOKEN_INGESTION.md

DSS Token Ingestion Guide

Complete documentation for multi-source token ingestion, merge strategies, and translation dictionaries.

Table of Contents

1. Overview
2. Supported Sources
3. Token Extraction
4. Merge System
5. Translation Dictionaries
6. MCP Tools
7. Examples

---

Overview

DSS provides a unified token ingestion pipeline that:

• Extracts tokens from multiple source formats (CSS, SCSS, Tailwind, JSON, Figma)
• Normalizes naming conventions to DSS canonical format
• Merges tokens with intelligent conflict resolution
• Generates translation dictionaries for traceability
• Exports to multiple output formats

┌─────────────────────────────────────────────────────────────────────┐
│                     DSS INGESTION PIPELINE                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   SOURCES              PROCESS                OUTPUT                 │
│   ───────              ───────                ──────                 │
│   CSS        ─┐                                                      │
│   SCSS       ─┼─→  Extract  →  Normalize  →  Merge  →  Export       │
│   Tailwind   ─┤       ↓           ↓           ↓          ↓          │
│   JSON/W3C   ─┤    Tokens    DSS Names    Unified    CSS/SCSS/      │
│   Figma      ─┘                                      TS/JSON         │
│                                                                      │
│                            ↓                                         │
│                   Translation Dictionary                             │
│                   (per-project mapping)                              │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

---

Supported Sources

CSS Custom Properties

Extracts CSS variables from :root and other selectors.

:root {
  /* Primary colors */
  --primary-500: #3B82F6;
  --primary-600: #2563EB;

  /* Spacing */
  --spacing-md: 16px;
}

Features:

• Comment extraction for descriptions
• Auto-detection of token type (color, dimension, etc.)
• Category inference from naming patterns

SCSS Variables

Extracts SCSS variables and maps.

// Brand colors
$primary-500: #3B82F6;
$primary-600: #2563EB;

// Spacing map
$spacing: (
  xs: 4px,
  sm: 8px,
  md: 16px,
  lg: 24px,
);

Features:

• Single variable extraction
• Map/object flattening
• Comment-based descriptions

Tailwind Configuration

Extracts tokens from tailwind.config.js/ts or Tailwind v4 CSS.

// tailwind.config.js
module.exports = {
  theme: {
    extend: {
      colors: {
        primary: {
          500: '#3B82F6',
          600: '#2563EB',
        },
      },
    },
  },
};

Features:

• Theme object parsing
• Extend section support
• Tailwind v4 @theme directive

JSON Token Files

Supports multiple JSON formats:

W3C Design Tokens:

{
  "color": {
    "primary": {
      "500": {
        "$value": "#3B82F6",
        "$type": "color",
        "$description": "Primary brand color"
      }
    }
  }
}

Style Dictionary:

{
  "color": {
    "primary": {
      "500": {
        "value": "#3B82F6",
        "comment": "Primary brand color"
      }
    }
  }
}

Tokens Studio (Figma Plugin):

{
  "global": {
    "color": {
      "primary": {
        "value": "#3B82F6",
        "type": "color"
      }
    }
  }
}

---

Token Extraction

Basic Usage

from tools.ingest import CSSTokenSource, SCSSTokenSource, JSONTokenSource

# Extract from CSS
css_source = CSSTokenSource()
css_tokens = await css_source.extract("/path/to/tokens.css")

# Extract from SCSS
scss_source = SCSSTokenSource()
scss_tokens = await scss_source.extract("/path/to/variables.scss")

# Extract from JSON
json_source = JSONTokenSource()
json_tokens = await json_source.extract("/path/to/tokens.json")

Token Structure

Each extracted token contains:

@dataclass
class DesignToken:
    name: str              # Normalized name: "color.primary.500"
    value: Any             # Token value: "#3B82F6"
    type: TokenType        # Type enum: COLOR, DIMENSION, etc.
    description: str       # From comments/descriptions
    source: str            # Source identifier: "css:tokens.css:12"
    source_file: str       # Original file path
    source_line: int       # Line number in source
    original_name: str     # Original name: "--primary-500"
    original_value: str    # Original value before processing
    category: TokenCategory # Category: COLORS, SPACING, etc.
    tags: List[str]        # Custom tags
    deprecated: bool       # Deprecation flag
    version: str           # Version string

Name Normalization

All source names are normalized to DSS canonical format:

Source Format	Original Name	DSS Canonical
CSS	--primary-500	primary.500
CSS (prefixed)	--heroui-primary-500	heroui.primary.500
SCSS	$primary-500	primary.500
camelCase	--brandAccent	brand.accent
SCREAMING	--SPACING_LG	spacing.lg

---

Merge System

Merge Strategies

Strategy	Description	Best For
FIRST	Keep first occurrence	Preserving original values
LAST	Use latest value (override)	Latest source wins
PREFER_FIGMA	Prioritize Figma sources	Design-led workflow
PREFER_CODE	Prioritize CSS/SCSS sources	Code-led workflow
PREFER_SPECIFIC	Prefer concrete values over var()	Resolving references
MERGE_METADATA	Combine metadata, use latest value	Preserving history
ERROR	Raise error on conflict	Strict validation
INTERACTIVE	Require user decision	Manual review

Usage

from tools.ingest import TokenMerger, MergeStrategy

# Create merger with strategy
merger = TokenMerger(strategy=MergeStrategy.PREFER_FIGMA)

# Merge multiple collections
result = merger.merge([css_tokens, scss_tokens, figma_tokens])

# Access results
print(f"Total tokens: {result.stats['total_tokens']}")
print(f"Conflicts resolved: {result.stats['conflicts_resolved']}")

# Examine conflicts
for conflict in result.conflicts:
    print(f"Token: {conflict.token_name}")
    print(f"  Source A: {conflict.existing.value} ({conflict.existing.source})")
    print(f"  Source B: {conflict.incoming.value} ({conflict.incoming.source})")
    print(f"  Resolution: {conflict.resolution}")
    print(f"  Final value: {conflict.resolved_token.value}")

Custom Resolver

def custom_resolver(conflict: MergeConflict) -> DesignToken:
    """Custom conflict resolution logic."""
    # Always prefer hex colors over named colors
    if conflict.incoming.value.startswith('#'):
        return conflict.incoming
    return conflict.existing

merger = TokenMerger(
    strategy=MergeStrategy.LAST,
    custom_resolver=custom_resolver
)

Merge Result

@dataclass
class MergeResult:
    collection: TokenCollection  # Merged tokens
    conflicts: List[MergeConflict]  # All conflicts encountered
    stats: Dict[str, int]  # Statistics
    warnings: List[str]  # Warning messages

Statistics include:

• total_tokens: Final token count
• new_tokens: Tokens added without conflict
• updated_tokens: Tokens updated during merge
• conflicts_resolved: Conflicts auto-resolved
• conflicts_unresolved: Conflicts requiring review

---

Translation Dictionaries

Purpose

Translation dictionaries map external token systems TO DSS canonical structure:

1. Traceability - Know where each token came from
2. Reproducibility - Re-run ingestion with same mappings
3. Documentation - Human-readable mapping reference
4. Custom Props - Track client-specific extensions

Schema

{
  "$schema": "dss-translation-v1",
  "project": "project-name",
  "source": "heroui | shadcn | css | scss | figma | tailwind",
  "version": "1.0.0",
  "created": "2025-01-15",
  "mappings": {
    "tokens": {
      "<source-token>": "<dss-canonical-token>"
    },
    "components": {
      "<source-component>": "<dss-component>[variant=value]"
    }
  },
  "custom_props": {
    "<dss-namespaced-token>": "<value>"
  },
  "unmapped": ["tokens that couldn't be mapped"],
  "notes": ["human-readable notes"]
}

Project Structure

project-acme/
├── .dss/
│   ├── config.json           # Project configuration
│   └── translations/
│       ├── heroui.json       # HeroUI → DSS mappings
│       ├── legacy-css.json   # Legacy CSS → DSS mappings
│       └── custom.json       # Custom props specific to ACME

---

MCP Tools

DSS provides MCP tools for token ingestion:

ingest_css_tokens

Extract tokens from CSS file.

ingest_css_tokens(source: "/path/to/tokens.css")

ingest_scss_tokens

Extract tokens from SCSS file.

ingest_scss_tokens(source: "/path/to/variables.scss")

ingest_tailwind_tokens

Extract tokens from Tailwind config.

ingest_tailwind_tokens(source: "/path/to/tailwind.config.js")

ingest_json_tokens

Extract tokens from JSON file (W3C, Style Dictionary, Tokens Studio).

ingest_json_tokens(source: "/path/to/tokens.json")

merge_tokens

Merge multiple token sources.

merge_tokens(
  sources: "/path/a.css,/path/b.scss",
  strategy: "prefer_figma"
)

export_tokens

Export tokens to various formats.

export_tokens(
  source: "/path/to/tokens.css",
  format: "typescript",  # css, scss, typescript, json, tailwind
  output_path: "/path/to/output.ts"
)

validate_tokens

Validate token collection.

validate_tokens(source: "/path/to/tokens.css")

---

Examples

Example 1: Merge HeroUI + Legacy CSS

import asyncio
from tools.ingest import (
    CSSTokenSource, TokenMerger, MergeStrategy
)

async def merge_heroui_legacy():
    css_source = CSSTokenSource()

    # Extract from both sources
    heroui_tokens = await css_source.extract("heroui-theme.css")
    legacy_tokens = await css_source.extract("legacy-styles.css")

    # Merge with HeroUI priority (newer system)
    merger = TokenMerger(strategy=MergeStrategy.LAST)
    result = merger.merge([legacy_tokens, heroui_tokens])

    print(f"Merged {result.stats['total_tokens']} tokens")
    print(f"Resolved {len(result.conflicts)} conflicts")

    # Export to TypeScript
    print(result.collection.to_typescript())

    return result

asyncio.run(merge_heroui_legacy())

Example 2: Build Translation Dictionary

import json
from tools.ingest import CSSTokenSource

async def build_translation():
    css_source = CSSTokenSource()
    tokens = await css_source.extract("heroui-theme.css")

    translation = {
        "$schema": "dss-translation-v1",
        "project": "heroui-migration",
        "source": "heroui",
        "mappings": {
            "tokens": {}
        }
    }

    # Build mappings
    for token in tokens.tokens:
        # Map HeroUI naming to DSS canonical
        dss_name = token.normalize_name()

        # Convert: heroui.primary.500 → color.primary.500
        if "primary" in dss_name or "secondary" in dss_name:
            dss_name = f"color.{dss_name.replace('heroui.', '')}"

        translation["mappings"]["tokens"][token.original_name] = dss_name

    # Save translation dictionary
    with open(".dss/translations/heroui.json", "w") as f:
        json.dump(translation, f, indent=2)

    return translation

Example 3: Diff Two Token Collections

from tools.ingest import CSSTokenSource
from tools.ingest.merge import TokenDiff

async def compare_versions():
    css_source = CSSTokenSource()

    old_tokens = await css_source.extract("tokens-v1.css")
    new_tokens = await css_source.extract("tokens-v2.css")

    diff = TokenDiff.diff(old_tokens, new_tokens)

    print(TokenDiff.summary(diff))
    # Output:
    # Token Diff Summary:
    # ========================================
    #
    # + Added (5):
    #   + color.accent.500: #F59E0B
    #   + color.accent.600: #D97706
    #   ...
    #
    # - Removed (2):
    #   - color.deprecated.old: #FF0000
    #   ...
    #
    # ~ Changed (3):
    #   ~ color.primary.500: #3B82F6 → #2563EB
    #   ...
    #
    #   Unchanged: 120

asyncio.run(compare_versions())

---

Best Practices

1. Always Use Translation Dictionaries

Even for simple projects, maintain translation dictionaries for:

• Audit trail of token origins
• Reproducible migrations
• Team documentation

2. Choose Appropriate Merge Strategy

Scenario	Recommended Strategy
Design system update	LAST
Design-led project	PREFER_FIGMA
Code-led project	PREFER_CODE
Strict validation	ERROR
Complex migration	MERGE_METADATA

3. Isolate Custom Props

Never pollute DSS core with client-specific tokens:

{
  "custom_props": {
    "color.brand.acme.primary": "#1E40AF",
    "color.brand.acme.accent": "#F59E0B"
  }
}

4. Validate Before Production

Always run validation before deploying:

result = await validate_tokens("/path/to/tokens.css")
if result["issues"]:
    print("Validation failed!")
    for issue in result["issues"]:
        print(f"  - {issue}")

5. Version Translation Dictionaries

Include version in translation dictionaries and track changes:

{
  "version": "1.2.0",
  "created": "2025-01-15",
  "updated": "2025-01-20",
  "changelog": [
    "1.2.0: Added accent color mappings",
    "1.1.0: Updated primary scale",
    "1.0.0: Initial translation"
  ]
}