dss/.dss/INTERNAL_PLUGIN_MARKETPLACE.md

# Internal Plugin Marketplace Architecture

**Date**: 2025-12-06
**Status**: ✅ Analysis Complete - Server-Side Plugin Loading Recommended
**Use Case**: Internal team use only (no external distribution)
**Confidence**: Very High (95%+)
**Validated By**: Gemini 3 Pro Preview (6-step thinkdeep analysis + expert validation)

---

## Executive Summary

**Question**: How should we distribute plugins for internal DSS team use?

**Answer**: **Server-Side Plugin Directory** with auto-loading via `plugin_loader.py`

**Previous Analysis Corrected**: The GitHub-based marketplace analysis was for PUBLIC distribution. For **internal-only use**, we need a much simpler approach.

---

## Key Requirements (Clarified)

1. ✅ **Internal use only** - DSS is for company team members
2. ✅ **Own plugins only** - No external plugin installation needed
3. ✅ **Simple distribution** - Hosted on DSS server itself
4. ✅ **Works with REMOTE + LOCAL modes** - Same plugins for both scenarios
5. ✅ **Zero client installation** - Plugins auto-available when connected

---

## Recommended Architecture: Server-Side Plugin Loading

### Core Concept

Instead of "installing" plugins to developer machines, **all plugins run on the DSS MCP server** and are auto-loaded on server startup.

```
┌─────────────────────────────────────────────────┐
│    DSS Server (dss.overbits.luz.uy)             │
│                                                  │
│  ┌────────────────────────────────────────┐    │
│  │  /plugins/ (Server-Side Directory)     │    │
│  │  ├── network-logger/                   │    │
│  │  ├── performance-analyzer/             │    │
│  │  └── custom-workflow/                  │    │
│  └────────────────┬───────────────────────┘    │
│                   │                              │
│  ┌────────────────▼───────────────────────┐    │
│  │  plugin_loader.py                      │    │
│  │  - Scans /plugins/ on server startup   │    │
│  │  - Imports TOOLS from each plugin      │    │
│  │  - Registers with FastMCP dynamically  │    │
│  └────────────────┬───────────────────────┘    │
│                   │                              │
│  ┌────────────────▼───────────────────────┐    │
│  │  DSS MCP Server                        │    │
│  │  - Built-in tools (project, debug)     │    │
│  │  - Plugin tools (auto-discovered)      │    │
│  │  - Strategy Pattern (REMOTE/LOCAL)     │    │
│  └────────────────┬───────────────────────┘    │
└───────────────────┼──────────────────────────────┘
                    │
                    │ MCP Protocol
                    │
         ┌──────────▼──────────┐
         │  Developer Machine  │
         │  - Claude Code CLI  │
         │  - Connects to MCP  │
         │  - All tools ready  │
         └─────────────────────┘
```

### Why Server-Side?

**REMOTE Mode**:
- ✅ MCP server on dss.overbits.luz.uy
- ✅ Plugins execute server-side
- ✅ Access to Shadow State API
- ✅ No localhost access needed

**LOCAL Mode**:
- ✅ MCP server still on dss.overbits.luz.uy
- ✅ Plugins use Strategy Pattern
- ✅ LocalStrategy handles browser automation
- ⚠️ May need reverse tunnel for localhost access (future enhancement)

---

## Implementation Design

### 1. Directory Structure

```
/home/overbits/dss/
├── tools/dss_mcp/
│   ├── server.py (main MCP server)
│   ├── plugin_loader.py (NEW - auto-discovery)
│   ├── tools/
│   │   ├── project_tools.py (built-in)
│   │   └── debug_tools.py (built-in)
│   └── plugins/ (NEW - plugin directory)
│       ├── __init__.py
│       ├── README.md (plugin development guide)
│       ├── _template/ (copy-paste template)
│       │   ├── __init__.py
│       │   ├── tools.py
│       │   └── README.md
│       ├── network-logger/
│       │   ├── __init__.py (exports TOOLS)
│       │   ├── tools.py (implementation)
│       │   └── README.md
│       └── performance-analyzer/
│           ├── __init__.py
│           ├── tools.py
│           └── README.md
```

### 2. Plugin Contract

Every plugin MUST have an `__init__.py` that exports a `TOOLS` list.

**Example**: `tools/dss_mcp/plugins/network-logger/__init__.py`

```python
"""Network Logger Plugin - Captures browser network requests."""

from .tools import get_network_requests, analyze_network_waterfall

# Plugin Contract:
# - TOOLS: List of callables to register as MCP tools
# - RESOURCES: (Optional) List of resources
# - PROMPTS: (Optional) List of prompts

TOOLS = [
    get_network_requests,
    analyze_network_waterfall
]

# Optional metadata
__version__ = "1.0.0"
__author__ = "DSS Team"
__description__ = "Captures and analyzes browser network traffic"
```

**Example**: `tools/dss_mcp/plugins/network-logger/tools.py`

```python
"""Network Logger tool implementations."""

import logging
from typing import Dict, Any, List, Optional

logger = logging.getLogger(__name__)

async def get_network_requests(
    session_id: str,
    filter_type: str = "all"
) -> List[Dict[str, Any]]:
    """
    Get browser network requests from Shadow State.

    Args:
        session_id: Browser session ID
        filter_type: Filter by type (xhr, fetch, all)

    Returns:
        List of network request objects
    """
    # Implementation using RemoteStrategy or LocalStrategy
    # Can access Shadow State API via /api/browser-logs/{session_id}
    logger.info(f"Fetching network requests for session {session_id}")

    # TODO: Implement actual network request retrieval
    return []

async def analyze_network_waterfall(
    session_id: str
) -> Dict[str, Any]:
    """
    Analyze network request waterfall.

    Returns:
        Performance analysis with waterfall data
    """
    logger.info(f"Analyzing network waterfall for session {session_id}")

    # TODO: Implement waterfall analysis
    return {"status": "not_implemented"}
```

### 3. Plugin Loader with Deferred Registration Pattern

**File**: `tools/dss_mcp/plugin_loader.py`

```python
"""
Dynamic plugin loader for DSS MCP server.

Implements the "Deferred Registration" pattern for FastMCP:
- Plugins define callable functions (not decorated)
- Loader scans /plugins/ directory
- Loader imports and collects TOOLS from each plugin
- Server applies @mcp.tool() decorator at runtime
"""

import importlib
import pkgutil
import os
import logging
from pathlib import Path
from typing import List, Callable

logger = logging.getLogger(__name__)


class PluginLoader:
    """
    Loads plugins from a directory and collects their tools.

    Resiliency: If one plugin fails to load, it logs the error
    and continues with other plugins (doesn't crash the server).
    """

    def __init__(self, plugin_dir: str):
        """
        Initialize plugin loader.

        Args:
            plugin_dir: Absolute path to plugins directory
        """
        self.plugin_dir = plugin_dir
        self.loaded_tools: List[Callable] = []
        self.failed_plugins: List[str] = []

    def load_plugins(self) -> List[Callable]:
        """
        Scan plugin directory, import modules, and aggregate tools.

        Returns:
            List of callables ready to be decorated by FastMCP.
        """
        if not os.path.exists(self.plugin_dir):
            logger.warning(f"Plugin directory not found: {self.plugin_dir}")
            return []

        logger.info(f"Scanning for plugins in {self.plugin_dir}")

        # Iterate over subdirectories in plugins/
        for module_info in pkgutil.iter_modules([self.plugin_dir]):
            if module_info.ispkg:
                self._load_single_plugin(module_info.name)

        logger.info(
            f"Plugin loading complete: "
            f"{len(self.loaded_tools)} tools from "
            f"{len(self.loaded_tools) - len(self.failed_plugins)} plugins"
        )

        if self.failed_plugins:
            logger.warning(f"Failed to load plugins: {', '.join(self.failed_plugins)}")

        return self.loaded_tools

    def _load_single_plugin(self, plugin_name: str):
        """
        Load a single plugin by name.

        Args:
            plugin_name: Name of the plugin subdirectory
        """
        try:
            # Dynamic import: tools.dss_mcp.plugins.<plugin_name>
            # This assumes the server runs from project root
            module_path = f"tools.dss_mcp.plugins.{plugin_name}"
            module = importlib.import_module(module_path)

            # Check plugin contract
            if hasattr(module, "TOOLS") and isinstance(module.TOOLS, list):
                self.loaded_tools.extend(module.TOOLS)

                # Log plugin metadata if available
                version = getattr(module, "__version__", "unknown")
                description = getattr(module, "__description__", "")

                logger.info(
                    f"✓ Loaded plugin '{plugin_name}' v{version}: "
                    f"{len(module.TOOLS)} tools"
                )
                if description:
                    logger.info(f"  └─ {description}")
            else:
                logger.warning(
                    f"✗ Plugin '{plugin_name}' skipped: "
                    f"No 'TOOLS' list found in __init__.py"
                )
                self.failed_plugins.append(plugin_name)

        except Exception as e:
            # CRITICAL: Do not crash the server for a bad plugin
            logger.error(
                f"✗ Failed to load plugin '{plugin_name}': {str(e)}",
                exc_info=True
            )
            self.failed_plugins.append(plugin_name)
```

### 4. Server Integration

**File**: `tools/dss_mcp/server.py` (update)

```python
"""DSS MCP Server with dynamic plugin loading."""

import os
from mcp.server.fastmcp import FastMCP

# Import built-in tools
from .tools.project_tools import PROJECT_TOOLS
from .tools.debug_tools import DEBUG_TOOLS

# Import plugin loader
from .plugin_loader import PluginLoader

# Initialize FastMCP
mcp = FastMCP("DSS Core")

# Initialize Plugin Loader
plugin_path = os.path.join(os.path.dirname(__file__), "plugins")
loader = PluginLoader(plugin_path)

# Load plugins
discovered_tools = loader.load_plugins()

# Register all tools dynamically
# FastMCP requires us to apply the decorator manually
for tool_func in discovered_tools:
    # Apply @mcp.tool() decorator to each discovered function
    mcp.tool()(tool_func)

# Also register built-in tools
for tool_func in PROJECT_TOOLS + DEBUG_TOOLS:
    mcp.tool()(tool_func)

# ... rest of server startup ...
```

---

## Implementation Plan

### Phase 1: Plugin Loader (1 day) - 🔴 CRITICAL

**Files to create**:
1. `tools/dss_mcp/plugin_loader.py` (complete implementation above)
2. `tools/dss_mcp/plugins/__init__.py` (empty)
3. `tools/dss_mcp/plugins/README.md` (plugin development guide)

**Files to update**:
1. `tools/dss_mcp/server.py` (integrate plugin loader)

**Testing**:
```bash
# Create test plugin
mkdir -p tools/dss_mcp/plugins/test_plugin
cat > tools/dss_mcp/plugins/test_plugin/__init__.py << 'EOF'
async def hello_dss():
    """Test plugin tool."""
    return "Hello from test plugin!"

TOOLS = [hello_dss]
EOF

# Restart MCP server
sudo supervisorctl restart dss-mcp

# Verify plugin loaded (check logs)
tail -f /var/log/supervisor/dss-mcp.log
```

### Phase 2: Plugin Directory Structure (0.5 day)

**Create plugin template**:

`tools/dss_mcp/plugins/_template/__init__.py`:
```python
"""
Template Plugin - Copy this directory to create new plugins.

Steps:
1. Copy _template/ to your-plugin-name/
2. Update __init__.py with your plugin metadata
3. Implement your tools in tools.py
4. Export TOOLS list with your tool functions
5. Restart MCP server: sudo supervisorctl restart dss-mcp
"""

from .tools import example_tool

TOOLS = [
    example_tool
]

__version__ = "1.0.0"
__author__ = "Your Name"
__description__ = "Description of what your plugin does"
```

`tools/dss_mcp/plugins/_template/tools.py`:
```python
"""Plugin tool implementations."""

import logging

logger = logging.getLogger(__name__)

async def example_tool(input_param: str) -> dict:
    """
    Example tool function.

    Args:
        input_param: Description of parameter

    Returns:
        Result dictionary
    """
    logger.info(f"Example tool called with: {input_param}")

    # Your implementation here
    return {
        "status": "success",
        "result": f"Processed: {input_param}"
    }
```

### Phase 3: Example Plugins (1 day)

Create 2-3 real-world plugins:

1. **network-logger** - Captures network requests via Shadow State
2. **performance-analyzer** - Analyzes performance metrics
3. **workflow-helper** - Common workflow shortcuts

### Phase 4: Optional Discovery API (0.5 day)

Add to `tools/api/server.py`:

```python
@app.get("/api/plugins/list")
async def list_plugins():
    """List all available server-side plugins."""
    plugins_dir = Path("/home/overbits/dss/tools/dss_mcp/plugins")
    plugins = []

    for plugin_dir in plugins_dir.iterdir():
        if plugin_dir.is_dir() and not plugin_dir.name.startswith('_'):
            try:
                # Import to get metadata
                module_name = f"tools.dss_mcp.plugins.{plugin_dir.name}"
                module = importlib.import_module(module_name)

                plugins.append({
                    "id": plugin_dir.name,
                    "name": plugin_dir.name.replace('-', ' ').title(),
                    "version": getattr(module, '__version__', 'unknown'),
                    "author": getattr(module, '__author__', 'unknown'),
                    "description": getattr(module, '__description__', ''),
                    "tools_count": len(getattr(module, 'TOOLS', []))
                })
            except Exception as e:
                logger.error(f"Error reading plugin {plugin_dir.name}: {e}")

    return {"plugins": plugins}
```

---

## Developer Workflow

### Adding a New Plugin

```bash
# 1. Copy template
cd /home/overbits/dss/tools/dss_mcp/plugins
cp -r _template my-new-plugin

# 2. Edit plugin files
vim my-new-plugin/__init__.py
# Update metadata (__version__, __author__, __description__)
# Update TOOLS list

vim my-new-plugin/tools.py
# Implement your tool functions

# 3. Restart MCP server
sudo supervisorctl restart dss-mcp

# 4. Verify plugin loaded
tail -20 /var/log/supervisor/dss-mcp.log | grep "my-new-plugin"

# 5. Test from Claude Code
# All tools should now be available automatically!
```

### Using Plugins (Developer Perspective)

```python
# Developer connects to DSS MCP server
# All plugins auto-available, no installation needed!

# Example: Use network logger plugin
result = await get_network_requests(
    session_id="abc123",
    filter_type="xhr"
)

# Example: Analyze performance
analysis = await analyze_network_waterfall(session_id="abc123")
```

---

## Key Advantages

1. ✅ **Zero Client Setup** - No installation on developer machines
2. ✅ **Central Management** - Update plugins server-side, all devs get new version
3. ✅ **Instant Availability** - Connect to MCP → all plugins ready
4. ✅ **Team Consistency** - Everyone uses exact same toolset
5. ✅ **Simple Development** - Copy template, edit, restart server
6. ✅ **Works with REMOTE/LOCAL** - Plugins use Strategy Pattern
7. ✅ **Resilient** - Bad plugin doesn't crash server
8. ✅ **No External Dependencies** - Everything internal

---

## Security & Access Control

### For Internal Use:

- ✅ **No authentication** needed for plugin loading (trusted code)
- ✅ **File permissions** on `/plugins/` directory (dev team only)
- ✅ **Access control** at MCP server level (who can connect)
- ✅ **Code review** before adding to `/plugins/` (team process)

### Security Note:

Since this is **internal-only**:
- All code in `/plugins/` is trusted (written by team)
- No sandboxing needed (not running untrusted code)
- Simple file permissions sufficient (Unix permissions)

---

## Comparison: Previous vs Current Approach

| Aspect | GitHub Marketplace (Previous) | Server-Side Loading (Current) |
|--------|-------------------------------|-------------------------------|
| **Use Case** | Public distribution | Internal team use |
| **Installation** | Client downloads | Zero installation |
| **Updates** | Each dev updates | Server-side only |
| **Complexity** | High (GitHub integration) | Low (file-based) |
| **Implementation** | 5-7 days | 2-3 days |
| **Maintenance** | Ongoing (registry API) | Minimal (add files) |
| **Developer UX** | Multi-step install | Instant availability |
| **Consistency** | Version mismatches | Always same version |

---

## Future Enhancements

### Possible Additions (Not Required Initially):

1. **Hot Reloading** - Reload plugins without server restart
   - Use `importlib.reload()` with file watching
   - WATCH OUT: Python caching issues

2. **Plugin Dependencies** - Handle inter-plugin dependencies
   - Add `REQUIRES = ['other-plugin']` to contract
   - Load in dependency order

3. **Plugin Versioning** - Semantic versioning and compatibility
   - Check `__min_server_version__` before loading
   - Warn on incompatible plugins

4. **Local Proxy Agent** - For LOCAL mode localhost access
   - Lightweight agent on developer machine
   - Reverse tunnel or SSH forwarding
   - Enables LOCAL browser automation from server-side plugins

---

## Expert Validation Summary

From Gemini 3 Pro Preview analysis:

> "This is a robust architectural pivot. Moving from a hypothetical 'GitHub Marketplace' to a concrete **Server-Side Plugin Loader** significantly reduces complexity while solving the immediate requirement: extending DSS capabilities without managing client-side installations."

**Key Validation Points**:

1. ✅ **Deferred Registration Pattern** correctly identified for FastMCP
2. ✅ **Resiliency Core** with try/except to prevent bad plugins crashing server
3. ✅ **Plugin Contract** with TOOLS list export pattern
4. ✅ **Naming Collision Risk** noted (mitigated by team discipline)
5. ✅ **Dependency Risk** handled via try/except and logging
6. ✅ **Hot Reloading** correctly identified as risky (use supervisord restart)

---

## Next Steps

**Priority Order**:

1. **🔴 Phase 1** - Create `plugin_loader.py` (1 day)
2. **🟡 Phase 2** - Create plugin directory structure (0.5 day)
3. **🟢 Phase 3** - Create example plugins (1 day)
4. **🔵 Phase 4** - Optional discovery API (0.5 day)

**Total Effort**: 2-3 days for complete implementation

**Start With**: Phase 1 - implement and test plugin_loader.py with a single test plugin.

---

**Status**: ✅ Ready for Implementation
**Confidence**: Very High (95%+)
**Last Updated**: 2025-12-06