Requirements
- Target platform
- OpenClaw
- Install method
- Manual import
- Extraction
- Extract archive
- Prerequisites
- OpenClaw
- Primary doc
- SKILL.md
Tencent SkillHub Β· AI
Senior Data Engineer
Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, implementing data governance, or troubleshooting data issues.
skill openclawclawhub Free
Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, implementing data governance, or troubleshooting data issues.
β¬ 0 downloads β
0 stars Unverified but indexed
Install for OpenClaw
Known item issue.
This item's current download entry is known to bounce back to a listing or homepage instead of returning a package file.
Quick setup
- Open the source page and confirm the package flow manually.
- Review SKILL.md if you can obtain the files.
- Treat this source as manual setup until the download is verified.
Package facts
- Download mode
- Manual review
- Package format
- ZIP package
- Source platform
- Tencent SkillHub
- What's included
- SKILL.md, references/data_modeling_patterns.md, references/data_pipeline_architecture.md, references/dataops_best_practices.md, references/troubleshooting.md, references/workflows.md
Validation
- Open the source listing and confirm there is a real package or setup artifact available.
- Review SKILL.md before asking your agent to continue.
- Treat this source as manual setup until the upstream download flow is fixed.
Install with your agent
Agent handoff
Use the source page and any available docs to guide the install because the item currently does not return a direct package file.
- Open the source page via Open source listing.
- If you can obtain the package, extract it into a folder your agent can access.
- Paste one of the prompts below and point your agent at the source page and extracted files.
New install
I tried to install a skill package from Yavira, but the item currently does not return a direct package file. Inspect the source page and any extracted docs, then tell me what you can confirm and any manual steps still required.
Upgrade existing
I tried to upgrade a skill package from Yavira, but the item currently does not return a direct package file. Compare the source page and any extracted docs with my current installation, then summarize what changed and what manual follow-up I still need.
Trust & source
Release facts
- Source
- Tencent SkillHub
- Verification
- Indexed source record
- Version
- 2.1.1
Provenance
- Publisher
- alirezarezvani
- Source page
- View original listing
- Canonical URL
- Open canonical page
Documentation
Senior Data Engineer
Production-grade data engineering skill for building scalable, reliable data systems.
Table of Contents
Trigger Phrases Quick Start Workflows Building a Batch ETL Pipeline Implementing Real-Time Streaming Data Quality Framework Setup Architecture Decision Framework Tech Stack Reference Documentation Troubleshooting
Trigger Phrases
Activate this skill when you see: Pipeline Design: "Design a data pipeline for..." "Build an ETL/ELT process..." "How should I ingest data from..." "Set up data extraction from..." Architecture: "Should I use batch or streaming?" "Lambda vs Kappa architecture" "How to handle late-arriving data" "Design a data lakehouse" Data Modeling: "Create a dimensional model..." "Star schema vs snowflake" "Implement slowly changing dimensions" "Design a data vault" Data Quality: "Add data validation to..." "Set up data quality checks" "Monitor data freshness" "Implement data contracts" Performance: "Optimize this Spark job" "Query is running slow" "Reduce pipeline execution time" "Tune Airflow DAG"
Core Tools
# Generate pipeline orchestration config python scripts/pipeline_orchestrator.py generate \ --type airflow \ --source postgres \ --destination snowflake \ --schedule "0 5 * * *" # Validate data quality python scripts/data_quality_validator.py validate \ --input data/sales.parquet \ --schema schemas/sales.json \ --checks freshness,completeness,uniqueness # Optimize ETL performance python scripts/etl_performance_optimizer.py analyze \ --query queries/daily_aggregation.sql \ --engine spark \ --recommend
Workflows
β See references/workflows.md for details
Architecture Decision Framework
Use this framework to choose the right approach for your data pipeline.
Batch vs Streaming
CriteriaBatchStreamingLatency requirementHours to daysSeconds to minutesData volumeLarge historical datasetsContinuous event streamsProcessing complexityComplex transformations, MLSimple aggregations, filteringCost sensitivityMore cost-effectiveHigher infrastructure costError handlingEasier to reprocessRequires careful design Decision Tree: Is real-time insight required? βββ Yes β Use streaming β βββ Is exactly-once semantics needed? β βββ Yes β Kafka + Flink/Spark Structured Streaming β βββ No β Kafka + consumer groups βββ No β Use batch βββ Is data volume > 1TB daily? βββ Yes β Spark/Databricks βββ No β dbt + warehouse compute
Lambda vs Kappa Architecture
AspectLambdaKappaComplexityTwo codebases (batch + stream)Single codebaseMaintenanceHigher (sync batch/stream logic)LowerReprocessingNative batch layerReplay from sourceUse caseML training + real-time servingPure event-driven When to choose Lambda: Need to train ML models on historical data Complex batch transformations not feasible in streaming Existing batch infrastructure When to choose Kappa: Event-sourced architecture All processing can be expressed as stream operations Starting fresh without legacy systems
Data Warehouse vs Data Lakehouse
FeatureWarehouse (Snowflake/BigQuery)Lakehouse (Delta/Iceberg)Best forBI, SQL analyticsML, unstructured dataStorage costHigher (proprietary format)Lower (open formats)FlexibilitySchema-on-writeSchema-on-readPerformanceExcellent for SQLGood, improvingEcosystemMature BI toolsGrowing ML tooling
Tech Stack
CategoryTechnologiesLanguagesPython, SQL, ScalaOrchestrationAirflow, Prefect, DagsterTransformationdbt, Spark, FlinkStreamingKafka, Kinesis, Pub/SubStorageS3, GCS, Delta Lake, IcebergWarehousesSnowflake, BigQuery, Redshift, DatabricksQualityGreat Expectations, dbt tests, Monte CarloMonitoringPrometheus, Grafana, Datadog
1. Data Pipeline Architecture
See references/data_pipeline_architecture.md for: Lambda vs Kappa architecture patterns Batch processing with Spark and Airflow Stream processing with Kafka and Flink Exactly-once semantics implementation Error handling and dead letter queues
2. Data Modeling Patterns
See references/data_modeling_patterns.md for: Dimensional modeling (Star/Snowflake) Slowly Changing Dimensions (SCD Types 1-6) Data Vault modeling dbt best practices Partitioning and clustering
3. DataOps Best Practices
See references/dataops_best_practices.md for: Data testing frameworks Data contracts and schema validation CI/CD for data pipelines Observability and lineage Incident response
Troubleshooting
β See references/troubleshooting.md for details
Category context
Agent frameworks, memory systems, reasoning layers, and model-native orchestration.
Source: Tencent SkillHub
Largest current source with strong distribution and engagement signals.
Package contents
Included in package
6 Docs
- SKILL.md
- references/data_modeling_patterns.md
- references/data_pipeline_architecture.md
- references/dataops_best_practices.md
- references/troubleshooting.md
- references/workflows.md