agent-observability

# Agent Observability Plugin

Expert guidance for instrumenting AI agents in production. This Claude Code plugin provides best practices, hooks against anti-patterns, and ready-to-use templates for observing multi-agent systems and workflows.

## Features

- **14 Specialized Skills** - LLM tracing, tool calls, multi-agent coordination, cost tracking, prompt A/B testing, guardrails, decision tracing, and more
- **2 AI Agents** - Codebase analyzer and instrumentation reviewer
- **2 Commands** - `/instrument` for planning and `/audit` for assessment
- **7 Anti-Pattern Hooks** - Catch common mistakes before they ship
- **4 Production Templates** - Copy-paste instrumentation code
- **9 Framework Guides** - LangChain, LangGraph, Claude Agent SDK, CrewAI, AutoGen, Pydantic AI, etc.
- **10 Vendor Integrations** - Langfuse, LangSmith, Arize, OpenTelemetry, Sentry, Datadog, etc.

## Installation

```bash
# From Claude Code
/plugin install agent-observability@caleb-davis-plugins

# Development mode
claude --plugin-dir /path/to/agent-observability
```

## Quick Start

### Audit Existing Telemetry

```bash
/audit
```

Scans your codebase for:
- Existing observability SDKs
- Instrumentation coverage gaps
- Anti-patterns with file:line references
- Prioritized recommendations

### Generate Instrumentation Plan

```bash
/instrument
```

Creates a tiered implementation plan:
- **T0: Foundation** - SDK init, basic spans, error capture
- **T1: Core Tracing** - LLM calls, tool executions
- **T2: Context** - Token/cost tracking, user context
- **T3: Multi-Agent** - Parent-child spans, handoffs
- **T4: Evaluation** - Quality metrics, feedback loops

### Framework-Specific

```bash
/instrument langgraph --vendor=langfuse
/instrument langchain --vendor=langsmith
/instrument crewai
```

## Skills

| Skill | Priority | Triggers |
|-------|----------|----------|
| `instrumentation-planning` | P1 | "what should I measure", "observability strategy" |
| `llm-call-tracing` | P1 | "trace LLM calls", "token tracking" |
| `tool-call-tracking` | P1 | "instrument tools", "tool execution spans" |
| `multi-agent-coordination` | P1 | "multi-agent tracing", "agent handoffs" |
| `token-cost-tracking` | P1 | "track tokens", "cost monitoring" |
| `prompt-versioning` | P1 | "prompt A/B testing", "prompt versions", "compare prompts" |
| `guardrails-safety` | P1 | "guardrails", "safety checks", "PII detection" |
| `decision-tracing` | P1 | "agent decisions", "tool selection", "why did agent" |
| `production-eval-strategy` | P1 | "production evaluation", "sampling strategy", "regression detection" |
| `memory-rag-instrumentation` | P2 | "RAG tracing", "retrieval instrumentation" |
| `human-in-the-loop` | P2 | "human approval tracking", "feedback loops" |
| `error-retry-tracking` | P2 | "error tracking", "retry instrumentation" |
| `evaluation-quality` | P2 | "agent evaluation", "quality metrics" |
| `session-conversation-tracking` | P2 | "session tracking", "multi-turn tracing" |

## Anti-Patterns Detected

The plugin's hooks warn you about:

1. **Full prompt/response logging** - PII risk, storage explosion
2. **Missing parent spans** - Broken traces, orphaned operations
3. **Secrets in traces** - API keys, tokens in span attributes
4. **Blocking telemetry** - Sync calls in agent hot path
5. **High cardinality** - Dynamic values in span names
6. **No token tracking** - Cost blindness
7. **Missing error context** - Can't debug failures

## Supported Frameworks

| Framework | Detection | Guide |
|-----------|-----------|-------|
| LangChain | `from langchain` | `references/frameworks/langchain.md` |
| LangGraph | `from langgraph` | `references/frameworks/langgraph.md` |
| Claude Agent SDK | `from claude_agent_sdk` | `references/frameworks/claude-agent-sdk.md` |
| OpenAI Agents | `from openai` | `references/frameworks/openai-agents-sdk.md` |
| CrewAI | `from crewai` | `references/frameworks/crewai.md` |
| AutoGen | `from autogen` | `references/frameworks/autogen.md` |
| Semantic Kernel | `semantic_kernel` | `references/frameworks/semantic-kernel.md` |
| Haystack | `from haystack` | `references/frameworks/haystack.md` |
| Pydantic AI | `from pydantic_ai` | `references/frameworks/pydantic-ai.md` |

## Supported Vendors

| Vendor | Detection | Guide |
|--------|-----------|-------|
| Langfuse | `from langfuse` | `references/vendors/langfuse.md` |
| LangSmith | `LANGCHAIN_TRACING_V2` | `references/vendors/langsmith.md` |
| OpenTelemetry | `from opentelemetry` | `references/vendors/opentelemetry.md` |
| Arize Phoenix | `from phoenix` | `references/vendors/arize-phoenix.md` |
| Helicone | `HELICONE_API_KEY` | `references/vendors/helicone.md` |
| Braintrust | `from braintrust` | `references/vendors/braintrust.md` |
| W&B Weave | `import weave` | `references/vendors/weave.md` |
| Datadog LLM | `ddtrace.llmobs` | `references/vendors/datadog-llm.md` |
| Sentry | `sentry_sdk` | `references/vendors/sentry.md` |
| Vercel AI SDK | `import { generateText }` | `references/vendors/vercel-ai-sdk.md` |

## Templates

Ready-to-use instrumentation code:

- **`templates/llm-tracing.py`** - LLM call tracing with token/cost tracking
- **`templates/tool-tracing.py`** - Tool execution instrumentation
- **`templates/agent-run.py`** - Agent lifecycle tracing
- **`templates/cost-tracking.py`** - Cost aggregation and budgeting

## Directory Structure

```
agent-observability/
├── .claude-plugin/
│   ├── plugin.json           # Plugin manifest
│   └── marketplace.json      # Marketplace metadata
├── agents/
│   ├── codebase-analyzer.md  # Framework/SDK detection
│   └── instrumentation-reviewer.md  # Quality review
├── commands/
│   ├── instrument.md         # Generate instrumentation plan
│   └── audit.md              # Audit existing telemetry
├── skills/                   # 14 focused capabilities
│   ├── instrumentation-planning/
│   ├── llm-call-tracing/
│   ├── tool-call-tracking/
│   ├── multi-agent-coordination/
│   ├── token-cost-tracking/
│   ├── prompt-versioning/
│   ├── guardrails-safety/
│   ├── decision-tracing/
│   ├── production-eval-strategy/
│   ├── memory-rag-instrumentation/
│   ├── human-in-the-loop/
│   ├── error-retry-tracking/
│   ├── evaluation-quality/
│   └── session-conversation-tracking/
├── hooks/
│   └── hooks.json            # Anti-pattern detection
├── references/
│   ├── methodology/          # Tiers, JTBD
│   ├── frameworks/           # Framework guides
│   ├── vendors/              # Vendor integrations
│   ├── anti-patterns/        # What to avoid
│   └── detection/            # Framework/SDK detection
├── templates/                # Production code
│   ├── llm-tracing.py
│   ├── tool-tracing.py
│   ├── agent-run.py
│   └── cost-tracking.py
├── CLAUDE.md                 # Plugin instructions
└── README.md                 # This file
```

## Contributing

1. **New Framework Support**
   - Add `references/frameworks/{framework}.md`
   - Add hooks in `hooks/hooks.json`
   - Update detection in `references/detection/`

2. **New Vendor Integration**
   - Add `references/vendors/{vendor}.md`
   - Update detection in `references/detection/`

3. **New Skills**
   - Add `skills/{skill-name}/SKILL.md`
   - Update CLAUDE.md skill routing table

## License

MIT

## Author

Caleb Davis ([email protected])