Initial commit: Clean DSS implementation
Migrated from design-system-swarm with fresh git history.
Old project history preserved in /home/overbits/apps/design-system-swarm
Core components:
- MCP Server (Python FastAPI with mcp 1.23.1)
- Claude Plugin (agents, commands, skills, strategies, hooks, core)
- DSS Backend (dss-mvp1 - token translation, Figma sync)
- Admin UI (Node.js/React)
- Server (Node.js/Express)
- Storybook integration (dss-mvp1/.storybook)
Self-contained configuration:
- All paths relative or use DSS_BASE_PATH=/home/overbits/dss
- PYTHONPATH configured for dss-mvp1 and dss-claude-plugin
- .env file with all configuration
- Claude plugin uses ${CLAUDE_PLUGIN_ROOT} for portability
Migration completed: $(date)
🤖 Clean migration with full functionality preserved
This commit is contained in:
Digital Production Factory
571
docs/TOKEN_INGESTION.md
Normal file
571
docs/TOKEN_INGESTION.md
Normal file
@@ -0,0 +1,571 @@
|
||||
# DSS Token Ingestion Guide
|
||||
|
||||
Complete documentation for multi-source token ingestion, merge strategies, and translation dictionaries.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Supported Sources](#supported-sources)
|
||||
3. [Token Extraction](#token-extraction)
|
||||
4. [Merge System](#merge-system)
|
||||
5. [Translation Dictionaries](#translation-dictionaries)
|
||||
6. [MCP Tools](#mcp-tools)
|
||||
7. [Examples](#examples)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
DSS provides a unified token ingestion pipeline that:
|
||||
|
||||
- Extracts tokens from multiple source formats (CSS, SCSS, Tailwind, JSON, Figma)
|
||||
- Normalizes naming conventions to DSS canonical format
|
||||
- Merges tokens with intelligent conflict resolution
|
||||
- Generates translation dictionaries for traceability
|
||||
- Exports to multiple output formats
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ DSS INGESTION PIPELINE │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ SOURCES PROCESS OUTPUT │
|
||||
│ ─────── ─────── ────── │
|
||||
│ CSS ─┐ │
|
||||
│ SCSS ─┼─→ Extract → Normalize → Merge → Export │
|
||||
│ Tailwind ─┤ ↓ ↓ ↓ ↓ │
|
||||
│ JSON/W3C ─┤ Tokens DSS Names Unified CSS/SCSS/ │
|
||||
│ Figma ─┘ TS/JSON │
|
||||
│ │
|
||||
│ ↓ │
|
||||
│ Translation Dictionary │
|
||||
│ (per-project mapping) │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Supported Sources
|
||||
|
||||
### CSS Custom Properties
|
||||
|
||||
Extracts CSS variables from `:root` and other selectors.
|
||||
|
||||
```css
|
||||
:root {
|
||||
/* Primary colors */
|
||||
--primary-500: #3B82F6;
|
||||
--primary-600: #2563EB;
|
||||
|
||||
/* Spacing */
|
||||
--spacing-md: 16px;
|
||||
}
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Comment extraction for descriptions
|
||||
- Auto-detection of token type (color, dimension, etc.)
|
||||
- Category inference from naming patterns
|
||||
|
||||
### SCSS Variables
|
||||
|
||||
Extracts SCSS variables and maps.
|
||||
|
||||
```scss
|
||||
// Brand colors
|
||||
$primary-500: #3B82F6;
|
||||
$primary-600: #2563EB;
|
||||
|
||||
// Spacing map
|
||||
$spacing: (
|
||||
xs: 4px,
|
||||
sm: 8px,
|
||||
md: 16px,
|
||||
lg: 24px,
|
||||
);
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Single variable extraction
|
||||
- Map/object flattening
|
||||
- Comment-based descriptions
|
||||
|
||||
### Tailwind Configuration
|
||||
|
||||
Extracts tokens from `tailwind.config.js/ts` or Tailwind v4 CSS.
|
||||
|
||||
```javascript
|
||||
// tailwind.config.js
|
||||
module.exports = {
|
||||
theme: {
|
||||
extend: {
|
||||
colors: {
|
||||
primary: {
|
||||
500: '#3B82F6',
|
||||
600: '#2563EB',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Theme object parsing
|
||||
- Extend section support
|
||||
- Tailwind v4 `@theme` directive
|
||||
|
||||
### JSON Token Files
|
||||
|
||||
Supports multiple JSON formats:
|
||||
|
||||
**W3C Design Tokens:**
|
||||
```json
|
||||
{
|
||||
"color": {
|
||||
"primary": {
|
||||
"500": {
|
||||
"$value": "#3B82F6",
|
||||
"$type": "color",
|
||||
"$description": "Primary brand color"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Style Dictionary:**
|
||||
```json
|
||||
{
|
||||
"color": {
|
||||
"primary": {
|
||||
"500": {
|
||||
"value": "#3B82F6",
|
||||
"comment": "Primary brand color"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Tokens Studio (Figma Plugin):**
|
||||
```json
|
||||
{
|
||||
"global": {
|
||||
"color": {
|
||||
"primary": {
|
||||
"value": "#3B82F6",
|
||||
"type": "color"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Token Extraction
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from tools.ingest import CSSTokenSource, SCSSTokenSource, JSONTokenSource
|
||||
|
||||
# Extract from CSS
|
||||
css_source = CSSTokenSource()
|
||||
css_tokens = await css_source.extract("/path/to/tokens.css")
|
||||
|
||||
# Extract from SCSS
|
||||
scss_source = SCSSTokenSource()
|
||||
scss_tokens = await scss_source.extract("/path/to/variables.scss")
|
||||
|
||||
# Extract from JSON
|
||||
json_source = JSONTokenSource()
|
||||
json_tokens = await json_source.extract("/path/to/tokens.json")
|
||||
```
|
||||
|
||||
### Token Structure
|
||||
|
||||
Each extracted token contains:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class DesignToken:
|
||||
name: str # Normalized name: "color.primary.500"
|
||||
value: Any # Token value: "#3B82F6"
|
||||
type: TokenType # Type enum: COLOR, DIMENSION, etc.
|
||||
description: str # From comments/descriptions
|
||||
source: str # Source identifier: "css:tokens.css:12"
|
||||
source_file: str # Original file path
|
||||
source_line: int # Line number in source
|
||||
original_name: str # Original name: "--primary-500"
|
||||
original_value: str # Original value before processing
|
||||
category: TokenCategory # Category: COLORS, SPACING, etc.
|
||||
tags: List[str] # Custom tags
|
||||
deprecated: bool # Deprecation flag
|
||||
version: str # Version string
|
||||
```
|
||||
|
||||
### Name Normalization
|
||||
|
||||
All source names are normalized to DSS canonical format:
|
||||
|
||||
| Source Format | Original Name | DSS Canonical |
|
||||
|---------------|---------------|---------------|
|
||||
| CSS | `--primary-500` | `primary.500` |
|
||||
| CSS (prefixed) | `--heroui-primary-500` | `heroui.primary.500` |
|
||||
| SCSS | `$primary-500` | `primary.500` |
|
||||
| camelCase | `--brandAccent` | `brand.accent` |
|
||||
| SCREAMING | `--SPACING_LG` | `spacing.lg` |
|
||||
|
||||
---
|
||||
|
||||
## Merge System
|
||||
|
||||
### Merge Strategies
|
||||
|
||||
| Strategy | Description | Best For |
|
||||
|----------|-------------|----------|
|
||||
| `FIRST` | Keep first occurrence | Preserving original values |
|
||||
| `LAST` | Use latest value (override) | Latest source wins |
|
||||
| `PREFER_FIGMA` | Prioritize Figma sources | Design-led workflow |
|
||||
| `PREFER_CODE` | Prioritize CSS/SCSS sources | Code-led workflow |
|
||||
| `PREFER_SPECIFIC` | Prefer concrete values over `var()` | Resolving references |
|
||||
| `MERGE_METADATA` | Combine metadata, use latest value | Preserving history |
|
||||
| `ERROR` | Raise error on conflict | Strict validation |
|
||||
| `INTERACTIVE` | Require user decision | Manual review |
|
||||
|
||||
### Usage
|
||||
|
||||
```python
|
||||
from tools.ingest import TokenMerger, MergeStrategy
|
||||
|
||||
# Create merger with strategy
|
||||
merger = TokenMerger(strategy=MergeStrategy.PREFER_FIGMA)
|
||||
|
||||
# Merge multiple collections
|
||||
result = merger.merge([css_tokens, scss_tokens, figma_tokens])
|
||||
|
||||
# Access results
|
||||
print(f"Total tokens: {result.stats['total_tokens']}")
|
||||
print(f"Conflicts resolved: {result.stats['conflicts_resolved']}")
|
||||
|
||||
# Examine conflicts
|
||||
for conflict in result.conflicts:
|
||||
print(f"Token: {conflict.token_name}")
|
||||
print(f" Source A: {conflict.existing.value} ({conflict.existing.source})")
|
||||
print(f" Source B: {conflict.incoming.value} ({conflict.incoming.source})")
|
||||
print(f" Resolution: {conflict.resolution}")
|
||||
print(f" Final value: {conflict.resolved_token.value}")
|
||||
```
|
||||
|
||||
### Custom Resolver
|
||||
|
||||
```python
|
||||
def custom_resolver(conflict: MergeConflict) -> DesignToken:
|
||||
"""Custom conflict resolution logic."""
|
||||
# Always prefer hex colors over named colors
|
||||
if conflict.incoming.value.startswith('#'):
|
||||
return conflict.incoming
|
||||
return conflict.existing
|
||||
|
||||
merger = TokenMerger(
|
||||
strategy=MergeStrategy.LAST,
|
||||
custom_resolver=custom_resolver
|
||||
)
|
||||
```
|
||||
|
||||
### Merge Result
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class MergeResult:
|
||||
collection: TokenCollection # Merged tokens
|
||||
conflicts: List[MergeConflict] # All conflicts encountered
|
||||
stats: Dict[str, int] # Statistics
|
||||
warnings: List[str] # Warning messages
|
||||
```
|
||||
|
||||
Statistics include:
|
||||
- `total_tokens`: Final token count
|
||||
- `new_tokens`: Tokens added without conflict
|
||||
- `updated_tokens`: Tokens updated during merge
|
||||
- `conflicts_resolved`: Conflicts auto-resolved
|
||||
- `conflicts_unresolved`: Conflicts requiring review
|
||||
|
||||
---
|
||||
|
||||
## Translation Dictionaries
|
||||
|
||||
### Purpose
|
||||
|
||||
Translation dictionaries map external token systems TO DSS canonical structure:
|
||||
|
||||
1. **Traceability** - Know where each token came from
|
||||
2. **Reproducibility** - Re-run ingestion with same mappings
|
||||
3. **Documentation** - Human-readable mapping reference
|
||||
4. **Custom Props** - Track client-specific extensions
|
||||
|
||||
### Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "dss-translation-v1",
|
||||
"project": "project-name",
|
||||
"source": "heroui | shadcn | css | scss | figma | tailwind",
|
||||
"version": "1.0.0",
|
||||
"created": "2025-01-15",
|
||||
"mappings": {
|
||||
"tokens": {
|
||||
"<source-token>": "<dss-canonical-token>"
|
||||
},
|
||||
"components": {
|
||||
"<source-component>": "<dss-component>[variant=value]"
|
||||
}
|
||||
},
|
||||
"custom_props": {
|
||||
"<dss-namespaced-token>": "<value>"
|
||||
},
|
||||
"unmapped": ["tokens that couldn't be mapped"],
|
||||
"notes": ["human-readable notes"]
|
||||
}
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
project-acme/
|
||||
├── .dss/
|
||||
│ ├── config.json # Project configuration
|
||||
│ └── translations/
|
||||
│ ├── heroui.json # HeroUI → DSS mappings
|
||||
│ ├── legacy-css.json # Legacy CSS → DSS mappings
|
||||
│ └── custom.json # Custom props specific to ACME
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MCP Tools
|
||||
|
||||
DSS provides MCP tools for token ingestion:
|
||||
|
||||
### `ingest_css_tokens`
|
||||
Extract tokens from CSS file.
|
||||
```
|
||||
ingest_css_tokens(source: "/path/to/tokens.css")
|
||||
```
|
||||
|
||||
### `ingest_scss_tokens`
|
||||
Extract tokens from SCSS file.
|
||||
```
|
||||
ingest_scss_tokens(source: "/path/to/variables.scss")
|
||||
```
|
||||
|
||||
### `ingest_tailwind_tokens`
|
||||
Extract tokens from Tailwind config.
|
||||
```
|
||||
ingest_tailwind_tokens(source: "/path/to/tailwind.config.js")
|
||||
```
|
||||
|
||||
### `ingest_json_tokens`
|
||||
Extract tokens from JSON file (W3C, Style Dictionary, Tokens Studio).
|
||||
```
|
||||
ingest_json_tokens(source: "/path/to/tokens.json")
|
||||
```
|
||||
|
||||
### `merge_tokens`
|
||||
Merge multiple token sources.
|
||||
```
|
||||
merge_tokens(
|
||||
sources: "/path/a.css,/path/b.scss",
|
||||
strategy: "prefer_figma"
|
||||
)
|
||||
```
|
||||
|
||||
### `export_tokens`
|
||||
Export tokens to various formats.
|
||||
```
|
||||
export_tokens(
|
||||
source: "/path/to/tokens.css",
|
||||
format: "typescript", # css, scss, typescript, json, tailwind
|
||||
output_path: "/path/to/output.ts"
|
||||
)
|
||||
```
|
||||
|
||||
### `validate_tokens`
|
||||
Validate token collection.
|
||||
```
|
||||
validate_tokens(source: "/path/to/tokens.css")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Merge HeroUI + Legacy CSS
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from tools.ingest import (
|
||||
CSSTokenSource, TokenMerger, MergeStrategy
|
||||
)
|
||||
|
||||
async def merge_heroui_legacy():
|
||||
css_source = CSSTokenSource()
|
||||
|
||||
# Extract from both sources
|
||||
heroui_tokens = await css_source.extract("heroui-theme.css")
|
||||
legacy_tokens = await css_source.extract("legacy-styles.css")
|
||||
|
||||
# Merge with HeroUI priority (newer system)
|
||||
merger = TokenMerger(strategy=MergeStrategy.LAST)
|
||||
result = merger.merge([legacy_tokens, heroui_tokens])
|
||||
|
||||
print(f"Merged {result.stats['total_tokens']} tokens")
|
||||
print(f"Resolved {len(result.conflicts)} conflicts")
|
||||
|
||||
# Export to TypeScript
|
||||
print(result.collection.to_typescript())
|
||||
|
||||
return result
|
||||
|
||||
asyncio.run(merge_heroui_legacy())
|
||||
```
|
||||
|
||||
### Example 2: Build Translation Dictionary
|
||||
|
||||
```python
|
||||
import json
|
||||
from tools.ingest import CSSTokenSource
|
||||
|
||||
async def build_translation():
|
||||
css_source = CSSTokenSource()
|
||||
tokens = await css_source.extract("heroui-theme.css")
|
||||
|
||||
translation = {
|
||||
"$schema": "dss-translation-v1",
|
||||
"project": "heroui-migration",
|
||||
"source": "heroui",
|
||||
"mappings": {
|
||||
"tokens": {}
|
||||
}
|
||||
}
|
||||
|
||||
# Build mappings
|
||||
for token in tokens.tokens:
|
||||
# Map HeroUI naming to DSS canonical
|
||||
dss_name = token.normalize_name()
|
||||
|
||||
# Convert: heroui.primary.500 → color.primary.500
|
||||
if "primary" in dss_name or "secondary" in dss_name:
|
||||
dss_name = f"color.{dss_name.replace('heroui.', '')}"
|
||||
|
||||
translation["mappings"]["tokens"][token.original_name] = dss_name
|
||||
|
||||
# Save translation dictionary
|
||||
with open(".dss/translations/heroui.json", "w") as f:
|
||||
json.dump(translation, f, indent=2)
|
||||
|
||||
return translation
|
||||
```
|
||||
|
||||
### Example 3: Diff Two Token Collections
|
||||
|
||||
```python
|
||||
from tools.ingest import CSSTokenSource
|
||||
from tools.ingest.merge import TokenDiff
|
||||
|
||||
async def compare_versions():
|
||||
css_source = CSSTokenSource()
|
||||
|
||||
old_tokens = await css_source.extract("tokens-v1.css")
|
||||
new_tokens = await css_source.extract("tokens-v2.css")
|
||||
|
||||
diff = TokenDiff.diff(old_tokens, new_tokens)
|
||||
|
||||
print(TokenDiff.summary(diff))
|
||||
# Output:
|
||||
# Token Diff Summary:
|
||||
# ========================================
|
||||
#
|
||||
# + Added (5):
|
||||
# + color.accent.500: #F59E0B
|
||||
# + color.accent.600: #D97706
|
||||
# ...
|
||||
#
|
||||
# - Removed (2):
|
||||
# - color.deprecated.old: #FF0000
|
||||
# ...
|
||||
#
|
||||
# ~ Changed (3):
|
||||
# ~ color.primary.500: #3B82F6 → #2563EB
|
||||
# ...
|
||||
#
|
||||
# Unchanged: 120
|
||||
|
||||
asyncio.run(compare_versions())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Always Use Translation Dictionaries
|
||||
|
||||
Even for simple projects, maintain translation dictionaries for:
|
||||
- Audit trail of token origins
|
||||
- Reproducible migrations
|
||||
- Team documentation
|
||||
|
||||
### 2. Choose Appropriate Merge Strategy
|
||||
|
||||
| Scenario | Recommended Strategy |
|
||||
|----------|---------------------|
|
||||
| Design system update | `LAST` |
|
||||
| Design-led project | `PREFER_FIGMA` |
|
||||
| Code-led project | `PREFER_CODE` |
|
||||
| Strict validation | `ERROR` |
|
||||
| Complex migration | `MERGE_METADATA` |
|
||||
|
||||
### 3. Isolate Custom Props
|
||||
|
||||
Never pollute DSS core with client-specific tokens:
|
||||
|
||||
```json
|
||||
{
|
||||
"custom_props": {
|
||||
"color.brand.acme.primary": "#1E40AF",
|
||||
"color.brand.acme.accent": "#F59E0B"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Validate Before Production
|
||||
|
||||
Always run validation before deploying:
|
||||
|
||||
```python
|
||||
result = await validate_tokens("/path/to/tokens.css")
|
||||
if result["issues"]:
|
||||
print("Validation failed!")
|
||||
for issue in result["issues"]:
|
||||
print(f" - {issue}")
|
||||
```
|
||||
|
||||
### 5. Version Translation Dictionaries
|
||||
|
||||
Include version in translation dictionaries and track changes:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.2.0",
|
||||
"created": "2025-01-15",
|
||||
"updated": "2025-01-20",
|
||||
"changelog": [
|
||||
"1.2.0: Added accent color mappings",
|
||||
"1.1.0: Updated primary scale",
|
||||
"1.0.0: Initial translation"
|
||||
]
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user