CI/CD Pipeline

CI/CD Pipeline (GitHub Actions)

Set up and manage CI/CD pipelines using GitHub Actions. Covers workflow creation, testing, deployment, release automation, and debugging.

When to Use

Setting up automated testing on push/PR Creating deployment pipelines (staging, production) Automating releases with changelogs and tags Debugging failing CI workflows Setting up matrix builds for cross-platform testing Managing secrets and environment variables in CI Optimizing CI with caching and parallelism

Node.js project

# .github/workflows/ci.yml name: CI on: push: branches: [main] pull_request: branches: [main] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 cache: npm - run: npm ci - run: npm test - run: npm run lint

Python project

# .github/workflows/ci.yml name: CI on: push: branches: [main] pull_request: branches: [main] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" cache: pip - run: pip install -r requirements.txt - run: pytest - run: ruff check .

Go project

# .github/workflows/ci.yml name: CI on: push: branches: [main] pull_request: branches: [main] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-go@v5 with: go-version: "1.22" - run: go test ./... - run: go vet ./...

Rust project

# .github/workflows/ci.yml name: CI on: push: branches: [main] pull_request: branches: [main] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable - uses: Swatinem/rust-cache@v2 - run: cargo test - run: cargo clippy -- -D warnings

Matrix builds (test across versions/OSes)

jobs: test: strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] node-version: [18, 20, 22] runs-on: ${{ matrix.os }} steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ matrix.node-version }} - run: npm ci - run: npm test

Conditional jobs

jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm test deploy: needs: test if: github.ref == 'refs/heads/main' && github.event_name == 'push' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: ./deploy.sh

Caching dependencies

• # Node.js (automatic with setup-node)
• uses: actions/setup-node@v4
• with:
• node-version: 20
• cache: npm # or yarn, pnpm
• # Generic caching
• uses: actions/cache@v4
• with:
• path: |
• ~/.cache/pip
• ~/.cargo/registry
• node_modules
• key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}
• restore-keys: |
• ${{ runner.os }}-deps-

Artifacts (save build outputs)

• uses: actions/upload-artifact@v4
• with:
• name: build-output
• path: dist/
• retention-days: 7
• # Download in another job
• uses: actions/download-artifact@v4
• with:
• name: build-output
• path: dist/

Run on schedule (cron)

on: schedule: - cron: "0 6 * * 1" # Every Monday at 6 AM UTC workflow_dispatch: # Also allow manual trigger

Deploy to production on tag

name: Release on: push: tags: - "v*" jobs: release: runs-on: ubuntu-latest permissions: contents: write steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 cache: npm - run: npm ci - run: npm run build - run: npm test # Create GitHub release - uses: softprops/action-gh-release@v2 with: generate_release_notes: true files: | dist/*.js dist/*.css

Deploy to multiple environments

name: Deploy on: push: branches: [main, staging] jobs: deploy: runs-on: ubuntu-latest environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'staging' }} steps: - uses: actions/checkout@v4 - run: npm ci && npm run build - run: | if [ "${{ github.ref }}" = "refs/heads/main" ]; then ./deploy.sh production else ./deploy.sh staging fi env: DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Docker build and push

name: Docker on: push: branches: [main] tags: ["v*"] jobs: build: runs-on: ubuntu-latest permissions: packages: write steps: - uses: actions/checkout@v4 - uses: docker/setup-buildx-action@v3 - uses: docker/login-action@v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - uses: docker/build-push-action@v6 with: push: true tags: | ghcr.io/${{ github.repository }}:latest ghcr.io/${{ github.repository }}:${{ github.sha }} cache-from: type=gha cache-to: type=gha,mode=max

npm publish on release

name: Publish on: release: types: [published] jobs: publish: runs-on: ubuntu-latest permissions: id-token: write steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 registry-url: https://registry.npmjs.org - run: npm ci - run: npm test - run: npm publish --provenance env: NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Set secrets via CLI

# Set a repository secret gh secret set DEPLOY_TOKEN --body "my-secret-value" # Set from a file gh secret set SSH_KEY < ~/.ssh/deploy_key # Set for a specific environment gh secret set DB_PASSWORD --env production --body "p@ssw0rd" # List secrets gh secret list # Delete a secret gh secret delete OLD_SECRET

Use secrets in workflows

env: # Available to all steps in this job DATABASE_URL: ${{ secrets.DATABASE_URL }} steps: - run: echo "Deploying..." env: # Available to this step only API_KEY: ${{ secrets.API_KEY }}

Environment protection rules

Set up via GitHub UI or API: Required reviewers before deployment Wait timers Branch restrictions Custom deployment branch policies # View environments gh api repos/{owner}/{repo}/environments | jq '.environments[].name'

Re-run failed jobs

# List recent workflow runs gh run list --limit 10 # View a specific run gh run view <run-id> # View failed job logs gh run view <run-id> --log-failed # Re-run failed jobs only gh run rerun <run-id> --failed # Re-run entire workflow gh run rerun <run-id>

Debug with SSH (using tmate)

• # Add this step before the failing step
• uses: mxschmitt/action-tmate@v3
• if: failure()
• with:
• limit-access-to-actor: true

Common failures and fixes

• "Permission denied" on scripts
• run: chmod +x ./scripts/deploy.sh && ./scripts/deploy.sh
• "Node modules not found"
• # Make sure npm ci runs before npm test
• run: npm ci # Install exact lockfile versions
• run: npm test # Now node_modules exists
• "Resource not accessible by integration"
• # Add permissions block
• permissions:
• contents: write
• packages: write
• pull-requests: write
• Cache not restoring
• # Check cache key matches - use hashFiles for lockfile
• key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
• # NOT: key: ${{ runner.os }}-node-${{ hashFiles('package.json') }}
• Workflow not triggering
• Check: is the workflow file on the default branch?
• Check: does the trigger event match? (push vs pull_request)
• Check: is the branch filter correct?
• # Manually trigger a workflow
• gh workflow run ci.yml --ref main

Validate locally before pushing

# Check YAML syntax python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" && echo "Valid" # Use actionlint (if installed) actionlint .github/workflows/ci.yml # Or via Docker docker run --rm -v "$(pwd):/repo" -w /repo rhysd/actionlint:latest

View workflow as graph

# List all workflows gh workflow list # View workflow definition gh workflow view ci.yml # Watch a running workflow gh run watch

Reusable workflows

# .github/workflows/reusable-test.yml name: Reusable Test on: workflow_call: inputs: node-version: required: false type: string default: "20" secrets: npm-token: required: false jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ inputs.node-version }} - run: npm ci - run: npm test # .github/workflows/ci.yml - caller name: CI on: [push, pull_request] jobs: test: uses: ./.github/workflows/reusable-test.yml with: node-version: "20"

Concurrency (prevent duplicate runs)

concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true # Cancel previous runs for same branch

Path filters (only run for relevant changes)

on: push: paths: - "src/**" - "package.json" - "package-lock.json" - ".github/workflows/ci.yml" paths-ignore: - "docs/**" - "*.md"

Monorepo: only test changed packages

jobs: changes: runs-on: ubuntu-latest outputs: api: ${{ steps.filter.outputs.api }} web: ${{ steps.filter.outputs.web }} steps: - uses: actions/checkout@v4 - uses: dorny/paths-filter@v3 id: filter with: filters: | api: - 'packages/api/**' web: - 'packages/web/**' test-api: needs: changes if: needs.changes.outputs.api == 'true' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: cd packages/api && npm ci && npm test test-web: needs: changes if: needs.changes.outputs.web == 'true' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: cd packages/web && npm ci && npm test

Tips

Use workflow_dispatch on every workflow for manual triggering during debugging Pin action versions to SHA for supply chain security: uses: actions/checkout@b4ffde... Use continue-on-error: true for non-critical steps (like linting) Set timeout-minutes on jobs to prevent runaway builds (default is 360 minutes) Use job outputs to pass data between jobs: outputs: result: ${{ steps.step-id.outputs.value }} For self-hosted runners: runs-on: self-hosted with labels for targeting specific machines