Senior Prompt Engineer

Senior Prompt Engineer

Prompt engineering patterns, LLM evaluation frameworks, and agentic system design.

Table of Contents

Quick Start Tools Overview Prompt Optimizer RAG Evaluator Agent Orchestrator Prompt Engineering Workflows Prompt Optimization Workflow Few-Shot Example Design Structured Output Design Reference Documentation Common Patterns Quick Reference

Quick Start

# Analyze and optimize a prompt file python scripts/prompt_optimizer.py prompts/my_prompt.txt --analyze # Evaluate RAG retrieval quality python scripts/rag_evaluator.py --contexts contexts.json --questions questions.json # Visualize agent workflow from definition python scripts/agent_orchestrator.py agent_config.yaml --visualize

1. Prompt Optimizer

Analyzes prompts for token efficiency, clarity, and structure. Generates optimized versions. Input: Prompt text file or string Output: Analysis report with optimization suggestions Usage: # Analyze a prompt file python scripts/prompt_optimizer.py prompt.txt --analyze # Output: # Token count: 847 # Estimated cost: $0.0025 (GPT-4) # Clarity score: 72/100 # Issues found: # - Ambiguous instruction at line 3 # - Missing output format specification # - Redundant context (lines 12-15 repeat lines 5-8) # Suggestions: # 1. Add explicit output format: "Respond in JSON with keys: ..." # 2. Remove redundant context to save 89 tokens # 3. Clarify "analyze" -> "list the top 3 issues with severity ratings" # Generate optimized version python scripts/prompt_optimizer.py prompt.txt --optimize --output optimized.txt # Count tokens for cost estimation python scripts/prompt_optimizer.py prompt.txt --tokens --model gpt-4 # Extract and manage few-shot examples python scripts/prompt_optimizer.py prompt.txt --extract-examples --output examples.json

2. RAG Evaluator

Evaluates Retrieval-Augmented Generation quality by measuring context relevance and answer faithfulness. Input: Retrieved contexts (JSON) and questions/answers Output: Evaluation metrics and quality report Usage: # Evaluate retrieval quality python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json # Output: # === RAG Evaluation Report === # Questions evaluated: 50 # # Retrieval Metrics: # Context Relevance: 0.78 (target: >0.80) # Retrieval Precision@5: 0.72 # Coverage: 0.85 # # Generation Metrics: # Answer Faithfulness: 0.91 # Groundedness: 0.88 # # Issues Found: # - 8 questions had no relevant context in top-5 # - 3 answers contained information not in context # # Recommendations: # 1. Improve chunking strategy for technical documents # 2. Add metadata filtering for date-sensitive queries # Evaluate with custom metrics python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json \ --metrics relevance,faithfulness,coverage # Export detailed results python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json \ --output report.json --verbose

3. Agent Orchestrator

Parses agent definitions and visualizes execution flows. Validates tool configurations. Input: Agent configuration (YAML/JSON) Output: Workflow visualization, validation report Usage: # Validate agent configuration python scripts/agent_orchestrator.py agent.yaml --validate # Output: # === Agent Validation Report === # Agent: research_assistant # Pattern: ReAct # # Tools (4 registered): # [OK] web_search - API key configured # [OK] calculator - No config needed # [WARN] file_reader - Missing allowed_paths # [OK] summarizer - Prompt template valid # # Flow Analysis: # Max depth: 5 iterations # Estimated tokens/run: 2,400-4,800 # Potential infinite loop: No # # Recommendations: # 1. Add allowed_paths to file_reader for security # 2. Consider adding early exit condition for simple queries # Visualize agent workflow (ASCII) python scripts/agent_orchestrator.py agent.yaml --visualize # Output: # ┌─────────────────────────────────────────┐ # │ research_assistant │ # │ (ReAct Pattern) │ # └─────────────────┬───────────────────────┘ # │ # ┌────────▼────────┐ # │ User Query │ # └────────┬────────┘ # │ # ┌────────▼────────┐ # │ Think │◄──────┐ # └────────┬────────┘ │ # │ │ # ┌────────▼────────┐ │ # │ Select Tool │ │ # └────────┬────────┘ │ # │ │ # ┌─────────────┼─────────────┐ │ # ▼ ▼ ▼ │ # [web_search] [calculator] [file_reader] # │ │ │ │ # └─────────────┼─────────────┘ │ # │ │ # ┌────────▼────────┐ │ # │ Observe │───────┘ # └────────┬────────┘ # │ # ┌────────▼────────┐ # │ Final Answer │ # └─────────────────┘ # Export workflow as Mermaid diagram python scripts/agent_orchestrator.py agent.yaml --visualize --format mermaid

Prompt Optimization Workflow

Use when improving an existing prompt's performance or reducing token costs. Step 1: Baseline current prompt python scripts/prompt_optimizer.py current_prompt.txt --analyze --output baseline.json Step 2: Identify issues Review the analysis report for: Token waste (redundant instructions, verbose examples) Ambiguous instructions (unclear output format, vague verbs) Missing constraints (no length limits, no format specification) Step 3: Apply optimization patterns IssuePattern to ApplyAmbiguous outputAdd explicit format specificationToo verboseExtract to few-shot examplesInconsistent resultsAdd role/persona framingMissing edge casesAdd constraint boundaries Step 4: Generate optimized version python scripts/prompt_optimizer.py current_prompt.txt --optimize --output optimized.txt Step 5: Compare results python scripts/prompt_optimizer.py optimized.txt --analyze --compare baseline.json # Shows: token reduction, clarity improvement, issues resolved Step 6: Validate with test cases Run both prompts against your evaluation set and compare outputs.

Few-Shot Example Design Workflow

Use when creating examples for in-context learning. Step 1: Define the task clearly Task: Extract product entities from customer reviews Input: Review text Output: JSON with {product_name, sentiment, features_mentioned} Step 2: Select diverse examples (3-5 recommended) Example TypePurposeSimple caseShows basic patternEdge caseHandles ambiguityComplex caseMultiple entitiesNegative caseWhat NOT to extract Step 3: Format consistently Example 1: Input: "Love my new iPhone 15, the camera is amazing!" Output: {"product_name": "iPhone 15", "sentiment": "positive", "features_mentioned": ["camera"]} Example 2: Input: "The laptop was okay but battery life is terrible." Output: {"product_name": "laptop", "sentiment": "mixed", "features_mentioned": ["battery life"]} Step 4: Validate example quality python scripts/prompt_optimizer.py prompt_with_examples.txt --validate-examples # Checks: consistency, coverage, format alignment Step 5: Test with held-out cases Ensure model generalizes beyond your examples.

Structured Output Design Workflow

• Use when you need reliable JSON/XML/structured responses.
• Step 1: Define schema
• {
• "type": "object",
• "properties": {
• "summary": {"type": "string", "maxLength": 200},
• "sentiment": {"enum": ["positive", "negative", "neutral"]},
• "confidence": {"type": "number", "minimum": 0, "maximum": 1}
• },
• "required": ["summary", "sentiment"]
• }
• Step 2: Include schema in prompt
• Respond with JSON matching this schema:
• summary (string, max 200 chars): Brief summary of the content
• sentiment (enum): One of "positive", "negative", "neutral"
• confidence (number 0-1): Your confidence in the sentiment
• Step 3: Add format enforcement
• IMPORTANT: Respond ONLY with valid JSON. No markdown, no explanation.
• Start your response with { and end with }
• Step 4: Validate outputs
• python scripts/prompt_optimizer.py structured_prompt.txt --validate-schema schema.json

Reference Documentation

FileContainsLoad when user asks aboutreferences/prompt_engineering_patterns.md10 prompt patterns with input/output examples"which pattern?", "few-shot", "chain-of-thought", "role prompting"references/llm_evaluation_frameworks.mdEvaluation metrics, scoring methods, A/B testing"how to evaluate?", "measure quality", "compare prompts"references/agentic_system_design.mdAgent architectures (ReAct, Plan-Execute, Tool Use)"build agent", "tool calling", "multi-agent"

Common Patterns Quick Reference

PatternWhen to UseExampleZero-shotSimple, well-defined tasks"Classify this email as spam or not spam"Few-shotComplex tasks, consistent format neededProvide 3-5 examples before the taskChain-of-ThoughtReasoning, math, multi-step logic"Think step by step..."Role PromptingExpertise needed, specific perspective"You are an expert tax accountant..."Structured OutputNeed parseable JSON/XMLInclude schema + format enforcement

Common Commands

# Prompt Analysis python scripts/prompt_optimizer.py prompt.txt --analyze # Full analysis python scripts/prompt_optimizer.py prompt.txt --tokens # Token count only python scripts/prompt_optimizer.py prompt.txt --optimize # Generate optimized version # RAG Evaluation python scripts/rag_evaluator.py --contexts ctx.json --questions q.json # Evaluate python scripts/rag_evaluator.py --contexts ctx.json --compare baseline # Compare to baseline # Agent Development python scripts/agent_orchestrator.py agent.yaml --validate # Validate config python scripts/agent_orchestrator.py agent.yaml --visualize # Show workflow python scripts/agent_orchestrator.py agent.yaml --estimate-cost # Token estimation