# Send Web Scraper as a Service to your agent
Hand the extracted package to your coding agent with a concrete install brief instead of figuring it out manually.
## Fast path
- Download the package from Yavira.
- Extract it into a folder your agent can access.
- Paste one of the prompts below and point your agent at the extracted folder.
## Suggested prompts
### New install

```text
I downloaded a skill package from Yavira. Read SKILL.md from the extracted folder and install it by following the included instructions. Tell me what you changed and call out any manual steps you could not complete.
```
### Upgrade existing

```text
I downloaded an updated skill package from Yavira. Read SKILL.md from the extracted folder, compare it with my current installation, and upgrade it while preserving any custom configuration unless the package docs explicitly say otherwise. Summarize what changed and any follow-up checks I should run.
```
## Machine-readable fields
```json
{
  "schemaVersion": "1.0",
  "item": {
    "slug": "web-scraper-as-a-service",
    "name": "Web Scraper as a Service",
    "source": "tencent",
    "type": "skill",
    "category": "开发工具",
    "sourceUrl": "https://clawhub.ai/seanwyngaard/web-scraper-as-a-service",
    "canonicalUrl": "https://clawhub.ai/seanwyngaard/web-scraper-as-a-service",
    "targetPlatform": "OpenClaw"
  },
  "install": {
    "downloadUrl": "/downloads/web-scraper-as-a-service",
    "sourceDownloadUrl": "https://wry-manatee-359.convex.site/api/v1/download?slug=web-scraper-as-a-service",
    "sourcePlatform": "tencent",
    "targetPlatform": "OpenClaw",
    "packageFormat": "ZIP package",
    "primaryDoc": "SKILL.md",
    "includedAssets": [
      "SKILL.md"
    ],
    "downloadMode": "redirect",
    "sourceHealth": {
      "source": "tencent",
      "status": "healthy",
      "reason": "direct_download_ok",
      "recommendedAction": "download",
      "checkedAt": "2026-05-07T17:22:31.273Z",
      "expiresAt": "2026-05-14T17:22:31.273Z",
      "httpStatus": 200,
      "finalUrl": "https://wry-manatee-359.convex.site/api/v1/download?slug=afrexai-annual-report",
      "contentType": "application/zip",
      "probeMethod": "head",
      "details": {
        "probeUrl": "https://wry-manatee-359.convex.site/api/v1/download?slug=afrexai-annual-report",
        "contentDisposition": "attachment; filename=\"afrexai-annual-report-1.0.0.zip\"",
        "redirectLocation": null,
        "bodySnippet": null
      },
      "scope": "source",
      "summary": "Source download looks usable.",
      "detail": "Yavira can redirect you to the upstream package for this source.",
      "primaryActionLabel": "Download for OpenClaw",
      "primaryActionHref": "/downloads/web-scraper-as-a-service"
    },
    "validation": {
      "installChecklist": [
        "Use the Yavira download entry.",
        "Review SKILL.md after the package is downloaded.",
        "Confirm the extracted package contains the expected setup assets."
      ],
      "postInstallChecks": [
        "Confirm the extracted package includes the expected docs or setup files.",
        "Validate the skill or prompts are available in your target agent workspace.",
        "Capture any manual follow-up steps the agent could not complete."
      ]
    }
  },
  "links": {
    "detailUrl": "https://openagent3.xyz/skills/web-scraper-as-a-service",
    "downloadUrl": "https://openagent3.xyz/downloads/web-scraper-as-a-service",
    "agentUrl": "https://openagent3.xyz/skills/web-scraper-as-a-service/agent",
    "manifestUrl": "https://openagent3.xyz/skills/web-scraper-as-a-service/agent.json",
    "briefUrl": "https://openagent3.xyz/skills/web-scraper-as-a-service/agent.md"
  }
}
```
## Documentation

### Web Scraper as a Service

Turn scraping briefs into deliverable scraping projects. Generates the scraper, runs it, cleans the data, and packages everything for the client.

### How to Use

/web-scraper-as-a-service "Scrape all products from example-store.com — need name, price, description, images. CSV output."
/web-scraper-as-a-service https://example.com --fields "title,price,rating,url" --format csv
/web-scraper-as-a-service brief.txt

### Step 1: Analyze the Target

Before writing any code:

Fetch the target URL to understand the page structure
Identify:

Is the site server-rendered (static HTML) or client-rendered (JavaScript/SPA)?
What anti-scraping measures are visible? (Cloudflare, CAPTCHAs, rate limits)
Pagination pattern (URL params, infinite scroll, load more button)
Data structure (product cards, table rows, list items)
Total estimated volume (number of pages/items)


Choose the right tool:

Static HTML → Python + requests + BeautifulSoup
JavaScript-rendered → Python + playwright
API available → Direct API calls (check network tab patterns)

### Step 2: Build the Scraper

Generate a complete Python script in scraper/ directory:

scraper/
  scrape.py           # Main scraper script
  requirements.txt    # Dependencies
  config.json         # Target URLs, fields, settings
  README.md           # Setup and usage instructions for client

scrape.py must include:

# Required features in every scraper:

# 1. Configuration
import json
config = json.load(open('config.json'))

# 2. Rate limiting (ALWAYS — be respectful)
import time
DELAY_BETWEEN_REQUESTS = 2  # seconds, adjustable in config

# 3. Retry logic
MAX_RETRIES = 3
RETRY_DELAY = 5

# 4. User-Agent rotation
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...",
    # ... at least 5 user agents
]

# 5. Progress tracking
print(f"Scraping page {current}/{total} — {items_collected} items collected")

# 6. Error handling
# - Log errors but don't crash on individual page failures
# - Save progress incrementally (don't lose data on crash)
# - Write errors to error_log.txt

# 7. Output
# - Save data incrementally (append to file, don't hold in memory)
# - Support CSV and JSON output
# - Clean and normalize data before saving

# 8. Resume capability
# - Track last successfully scraped page/URL
# - Can resume from where it left off if interrupted

### Step 3: Data Cleaning

After scraping, clean the data:

Remove duplicates (by unique identifier or composite key)
Normalize text (strip extra whitespace, fix encoding issues, consistent capitalization)
Validate data (no empty required fields, prices are numbers, URLs are valid)
Standardize formats (dates to ISO 8601, currency to numbers, consistent units)
Generate data quality report:
Data Quality Report
───────────────────
Total records: 2,487
Duplicates removed: 13
Empty fields filled: 0
Fields with issues: price (3 records had non-numeric values — cleaned)
Completeness: 99.5%

### Step 4: Client Deliverable Package

Generate a complete deliverable:

delivery/
  data.csv                    # Clean data in requested format
  data.json                   # JSON alternative
  data-quality-report.md      # Quality metrics
  scraper-documentation.md    # How the scraper works
  README.md                   # Quick start guide

scraper-documentation.md includes:

What was scraped and from where
How many records collected
Data fields and their descriptions
How to re-run the scraper
Known limitations
Date of scraping

### Step 5: Output to User

Present:

Summary: X records scraped from Y pages, Z% data quality
Sample data: First 5 rows of the output
File locations: Where the deliverables are saved
Client handoff notes: What to tell the client about the data

### Scraper Templates

Based on the target type, use the appropriate template:

### E-commerce Product Scraper

Fields: name, price, original_price, discount, description, images, category, sku, rating, review_count, availability, url

### Real Estate Listings

Fields: address, price, bedrooms, bathrooms, sqft, lot_size, listing_type, agent, description, images, url

### Job Listings

Fields: title, company, location, salary, job_type, description, requirements, posted_date, url

### Directory/Business Listings

Fields: business_name, address, phone, website, category, rating, review_count, hours, description

### News/Blog Articles

Fields: title, author, date, content, tags, url, image

### Ethical Scraping Rules

Always respect robots.txt — check before scraping
Rate limit — minimum 2 second delay between requests
Identify yourself — use realistic but honest User-Agent
Don't scrape personal data (emails, phone numbers) unless explicitly authorized by the client AND the data is publicly displayed
Cache responses — don't re-scrape pages unnecessarily
Check ToS — note if the site's terms prohibit scraping and inform the client
## Trust
- Source: tencent
- Verification: Indexed source record
- Publisher: seanwyngaard
- Version: 1.0.0
## Source health
- Status: healthy
- Source download looks usable.
- Yavira can redirect you to the upstream package for this source.
- Health scope: source
- Reason: direct_download_ok
- Checked at: 2026-05-07T17:22:31.273Z
- Expires at: 2026-05-14T17:22:31.273Z
- Recommended action: Download for OpenClaw
## Links
- [Detail page](https://openagent3.xyz/skills/web-scraper-as-a-service)
- [Send to Agent page](https://openagent3.xyz/skills/web-scraper-as-a-service/agent)
- [JSON manifest](https://openagent3.xyz/skills/web-scraper-as-a-service/agent.json)
- [Markdown brief](https://openagent3.xyz/skills/web-scraper-as-a-service/agent.md)
- [Download page](https://openagent3.xyz/downloads/web-scraper-as-a-service)