Why do MCP servers need different observability than regular APIs?

MCP servers are called by AI agents at runtime, which means traffic patterns are unpredictable and failures are hard to reproduce. An agent might call the same tool 200 times in a loop if its reasoning gets stuck, or call a tool with subtly wrong arguments that pass validation but return misleading data. Traditional API monitoring catches latency and errors. It misses AI-specific failure modes like reasoning loops, hallucinated arguments, and cascading tool failures.

What OpenTelemetry semantic conventions apply to MCP servers?

The OpenTelemetry GenAI semantic conventions (stabilized in early 2026) define standard attributes for AI workloads: gen_ai.system (the LLM provider), gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens. For MCP specifically, you should also track the MCP session ID, transport type (Streamable HTTP vs stdio), and whether the call was initiated by an agent or a human. These attributes let you correlate tool calls back to the originating conversation.

How do I detect an agent tool loop in production before it runs up my API bill?

Track tool call rate per session with a sliding window counter. If a single session calls the same tool more than N times in 5 minutes (start with N=10 and tune it), fire an alert and optionally circuit-break the session. Most legitimate workflows call a given tool 1-3 times per conversation. A spike to 50+ calls almost always means the agent is looping on a failed state.

Should I trace every MCP tool call or sample them?

Trace every tool call in development and staging. In production, use head-based sampling at 20-50% for normal traffic, but keep 100% coverage on error spans and slow calls (above your P95 baseline). Never sample tool failures -- the pattern of which tools fail, in what sequence, is exactly the signal you need to debug agent behavior. Storage costs are low compared to the debugging time saved.

How do I attribute MCP tool costs to specific agent features or conversations?

Add a cost attribute to every tool call span: the estimated token or API cost. Propagate your conversation ID, user ID, and feature flag context through the OpenTelemetry baggage API so every downstream span inherits them automatically. Then you can query your telemetry backend for total tool cost by conversation, by feature, or by agent version -- which tells you which agent behaviors are expensive to run.

What is the most common MCP observability mistake?

Treating MCP servers like REST APIs and only tracking HTTP status codes and latency. An MCP tool can return HTTP 200 with a response that causes the agent to hallucinate its next step. You need to track what the tool returned, not just whether it returned successfully. Log the tool output (with PII scrubbing) at debug level and sample it at info level so you can reconstruct what the agent saw during a debugging session.

MCP Servers in Production: Observability from Day One

Your MCP server is live. Tools are responding. The HTTP layer is healthy. But you have no idea what an agent actually sees when it calls get_account_balance, no idea why the same tool was called 47 times in one session yesterday, and no idea whether the answer your agent gave at 2 AM was based on stale data or fresh.

This is the observability gap that bites teams after their first MCP deployment. The server is running. The API is responding. But you're flying blind on what your agents are actually getting and doing with it.

This guide walks through instrumenting an MCP server for production from scratch: tracing tool calls end to end, detecting loops before they run up your bill, and building the dashboards that actually tell you what's happening.

Why MCP Needs Different Monitoring

Traditional API monitoring answers: did the request succeed, how fast was it, what was the HTTP status? That's enough for a REST API serving human users.

MCP servers don't serve humans. They serve AI agents at runtime -- and agents fail differently. An agent can call the same tool in a loop because its reasoning got stuck. It can receive a perfectly valid HTTP 200 with data that causes it to produce a wrong answer in the next step. It can make subtly malformed tool arguments that pass schema validation but return misleading results.

None of these show up in a standard APM dashboard. You need to see inside the tool calls.

The good news: OpenTelemetry's GenAI semantic conventions (stabilized in early 2026) give you the vocabulary to describe exactly what's happening in an MCP tool call. Once you're emitting the right spans and attributes, any standard observability backend -- Grafana, Datadog, New Relic, Honeycomb -- can query them.

Here's what matters at a glance:

Signal	Standard API	MCP Server
Latency	P50, P99 per endpoint	P50, P99 per tool + per agent
Errors	HTTP 4xx/5xx	HTTP errors + semantic tool failures
Volume	Requests/sec	Calls/session (loop detection)
Content	Optional	Critical (what did the agent get?)
Cost	N/A	Tokens + downstream API costs

Step 1: Set Up the OpenTelemetry SDK

Start with a minimal instrumentation setup. You'll add to it, but get the plumbing right first.

// mcp-server/instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { Resource } from '@opentelemetry/resources';
import {
  SEMRESATTRS_SERVICE_NAME,
  SEMRESATTRS_SERVICE_VERSION,
} from '@opentelemetry/semantic-conventions';
 
const resource = new Resource({
  [SEMRESATTRS_SERVICE_NAME]: 'mcp-tools-server',
  [SEMRESATTRS_SERVICE_VERSION]: process.env.SERVICE_VERSION ?? '0.0.0',
  'deployment.environment': process.env.NODE_ENV ?? 'development',
  'mcp.transport': process.env.MCP_TRANSPORT ?? 'streamable-http',
});
 
const traceExporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318/v1/traces',
});
 
const metricExporter = new OTLPMetricExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318/v1/metrics',
});
 
export const sdk = new NodeSDK({
  resource,
  traceExporter,
  metricReader: new PeriodicExportingMetricReader({
    exporter: metricExporter,
    exportIntervalMillis: 30_000,
  }),
});
 
// Call sdk.start() before importing anything else
sdk.start();

Initialize this before your MCP server starts:

// mcp-server/index.ts  (top of file, before other imports)
import './instrumentation';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
// ... rest of your server

Step 2: Instrument Tool Calls

The core of MCP observability is tracing individual tool calls. Each call should produce a span with the tool name, input arguments, output, duration, and any error.

// mcp-server/tracing.ts
import { trace, SpanStatusCode, context, propagation } from '@opentelemetry/api';
import { scrubPII } from './scrubbing';
 
const tracer = trace.getTracer('mcp-tools', '1.0.0');
 
interface ToolCallContext {
  toolName: string;
  sessionId: string;
  conversationId: string;
  callId: string;
  agentVersion?: string;
}
 
export async function traceToolCall<T>(
  ctx: ToolCallContext,
  args: Record<string, unknown>,
  handler: () => Promise<T>
): Promise<T> {
  return tracer.startActiveSpan(
    `mcp.tool.${ctx.toolName}`,
    {
      attributes: {
        // GenAI semantic conventions
        'gen_ai.tool.name': ctx.toolName,
        'gen_ai.tool.call.id': ctx.callId,
        'gen_ai.system': 'mcp',
 
        // MCP-specific
        'mcp.session.id': ctx.sessionId,
        'mcp.conversation.id': ctx.conversationId,
        'mcp.agent.version': ctx.agentVersion ?? 'unknown',
 
        // Sanitized input for debugging
        'mcp.tool.input': JSON.stringify(scrubPII(args)),
      }
    },
    async (span) => {
      const startTime = Date.now();
      try {
        const result = await handler();
        const duration = Date.now() - startTime;
 
        span.setAttributes({
          'mcp.tool.duration_ms': duration,
          'mcp.tool.success': true,
          // Log output at debug level only — too verbose for info
          'mcp.tool.output_preview': JSON.stringify(scrubPII(result as Record<string, unknown>)).slice(0, 200),
        });
 
        span.setStatus({ code: SpanStatusCode.OK });
        return result;
      } catch (error) {
        const duration = Date.now() - startTime;
        const err = error as Error;
 
        span.setAttributes({
          'mcp.tool.duration_ms': duration,
          'mcp.tool.success': false,
          'mcp.tool.error_type': err.constructor.name,
          'mcp.tool.error_message': err.message,
        });
 
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: err.message,
        });
        span.recordException(err);
 
        throw error;
      } finally {
        span.end();
      }
    }
  );
}

Use this wrapper around every tool in your MCP server:

// Example: account lookup tool
server.registerTool('get_account_balance', AccountBalanceSchema, async (args, callInfo) => {
  return traceToolCall(
    {
      toolName: 'get_account_balance',
      sessionId: callInfo.sessionId ?? 'unknown',
      conversationId: args._conversationId ?? 'unknown',
      callId: callInfo.callId ?? crypto.randomUUID(),
    },
    args,
    () => accountService.getBalance(args.accountId)
  );
});

Step 3: Track Metrics for Alerting

Spans give you detailed debugging data. Metrics give you the aggregates you need for alerting and dashboards.

// mcp-server/metrics.ts
import { metrics } from '@opentelemetry/api';
 
const meter = metrics.getMeter('mcp-tools', '1.0.0');
 
// Counter: total calls per tool
const toolCallCounter = meter.createCounter('mcp.tool.calls', {
  description: 'Total MCP tool calls',
  unit: '1',
});
 
// Histogram: call duration distribution
const toolDurationHistogram = meter.createHistogram('mcp.tool.duration', {
  description: 'MCP tool call duration in milliseconds',
  unit: 'ms',
  advice: { explicitBucketBoundaries: [50, 100, 250, 500, 1000, 2500, 5000] },
});
 
// Counter: errors per tool
const toolErrorCounter = meter.createCounter('mcp.tool.errors', {
  description: 'Total MCP tool call errors',
  unit: '1',
});
 
interface ToolMetricLabels {
  tool: string;
  session: string;
  error_type?: string;
}
 
export function recordToolCall(
  labels: ToolMetricLabels,
  durationMs: number,
  success: boolean
): void {
  const dimensions = {
    'tool.name': labels.tool,
    'mcp.session.id': labels.session,
  };
 
  toolCallCounter.add(1, dimensions);
  toolDurationHistogram.record(durationMs, dimensions);
 
  if (!success) {
    toolErrorCounter.add(1, {
      ...dimensions,
      'error.type': labels.error_type ?? 'unknown',
    });
  }
}

Step 4: Loop Detection

This is the alerting feature you'll wish you had from day one. Agents can get stuck calling the same tool repeatedly -- usually because a tool returned unexpected data and the agent retried instead of escalating.

// mcp-server/loop-detection.ts
interface SessionWindow {
  calls: Map<string, number[]>; // toolName -> timestamps
  lastAlert: number;
}
 
const sessionWindows = new Map<string, SessionWindow>();
const WINDOW_MS = 5 * 60 * 1000; // 5 minutes
const LOOP_THRESHOLD = 15;        // calls to same tool in window
 
export function trackCallForLoops(
  sessionId: string,
  toolName: string,
  onLoopDetected: (sessionId: string, toolName: string, callCount: number) => void
): void {
  const now = Date.now();
 
  if (!sessionWindows.has(sessionId)) {
    sessionWindows.set(sessionId, { calls: new Map(), lastAlert: 0 });
  }
 
  const window = sessionWindows.get(sessionId)!;
 
  if (!window.calls.has(toolName)) {
    window.calls.set(toolName, []);
  }
 
  const timestamps = window.calls.get(toolName)!;
 
  // Add current call and prune old timestamps
  timestamps.push(now);
  const cutoff = now - WINDOW_MS;
  const recentCalls = timestamps.filter(t => t > cutoff);
  window.calls.set(toolName, recentCalls);
 
  // Alert if threshold exceeded (throttle to once per 5 minutes per session)
  if (recentCalls.length >= LOOP_THRESHOLD && now - window.lastAlert > WINDOW_MS) {
    window.lastAlert = now;
    onLoopDetected(sessionId, toolName, recentCalls.length);
  }
}
 
// Cleanup stale sessions (run on a timer)
export function cleanupStaleSessions(maxAgeMs = 30 * 60 * 1000): void {
  const cutoff = Date.now() - maxAgeMs;
  for (const [sessionId, window] of sessionWindows.entries()) {
    const allTimestamps = Array.from(window.calls.values()).flat();
    const mostRecent = Math.max(...allTimestamps, 0);
    if (mostRecent < cutoff) {
      sessionWindows.delete(sessionId);
    }
  }
}

Wire this into your tool call handler:

server.registerTool('get_account_balance', AccountBalanceSchema, async (args, callInfo) => {
  const sessionId = callInfo.sessionId ?? 'unknown';
 
  trackCallForLoops(sessionId, 'get_account_balance', async (sid, tool, count) => {
    // Send to your alerting system
    await alert.send({
      severity: 'warning',
      message: `Loop detected: session ${sid} called ${tool} ${count}x in 5 minutes`,
      metadata: { sessionId: sid, toolName: tool, callCount: count },
    });
  });
 
  return traceToolCall(/* ... */);
});

Step 5: PII Scrubbing

You're logging tool inputs and outputs. Those almost certainly contain PII. Do the scrubbing before spans are exported.

// mcp-server/scrubbing.ts
const PII_PATTERNS: Array<[RegExp, string]> = [
  [/\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/g, '[CARD_NUMBER]'],
  [/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/gi, '[EMAIL]'],
  [/\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/g, '[PHONE]'],
  [/\b\d{3}-\d{2}-\d{4}\b/g, '[SSN]'],
];
 
export function scrubPII(data: unknown): unknown {
  if (typeof data === 'string') {
    return PII_PATTERNS.reduce((str, [pattern, replacement]) =>
      str.replace(pattern, replacement), data
    );
  }
 
  if (Array.isArray(data)) {
    return data.map(scrubPII);
  }
 
  if (data !== null && typeof data === 'object') {
    return Object.fromEntries(
      Object.entries(data as Record<string, unknown>).map(([k, v]) => [k, scrubPII(v)])
    );
  }
 
  return data;
}

This is minimal but covers the most common PII types. Extend with patterns for your specific domain.

How the Traces Connect

Here's how a single agent turn flows through the trace system, from the agent's reasoning step down to your MCP tool and back:

graph TD
  A[Agent: user turn received] --> B[LLM inference span]
  B --> C[Tool call decision]
  C --> D[MCP gateway span]
  D --> E[get_account_balance span]
  E --> F[DB query span]
  F --> G[Response to agent]
  G --> H[LLM inference span 2]
  H --> I[Final response]

Distributed trace spanning agent reasoning, MCP gateway, and tool server for a single conversation turn

Each box is a span. They all share the same trace ID. When something goes wrong -- say the DB query returns stale data -- you can pull the full trace and see exactly what the agent received, what it inferred, and what it said.

Multi-backend export

If you're already shipping traces to Grafana or Datadog, you can fan them out to multiple backends with SpanProcessor instances. The OpenTelemetry SDK supports as many exporters as you want — pick the pattern that matches your tooling.

// instrumentation.ts — multi-backend export using SimpleSpanProcessor
import { NodeSDK } from '@opentelemetry/sdk-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
 
const primaryExporter = new OTLPTraceExporter({
  url: process.env.PRIMARY_OTLP_ENDPOINT ?? 'http://localhost:4318/v1/traces',
});
 
// Additional export to a second backend (e.g. for retention or correlation)
const secondaryExporter = new OTLPTraceExporter({
  url: process.env.SECONDARY_OTLP_ENDPOINT ?? '',
  headers: { authorization: `Bearer ${process.env.SECONDARY_OTLP_TOKEN ?? ''}` },
});
 
export const sdk = new NodeSDK({
  resource: new Resource({ 'service.name': 'mcp-tools-server' }),
  spanProcessors: [
    new SimpleSpanProcessor(primaryExporter),
    new SimpleSpanProcessor(secondaryExporter),
  ],
});
 
sdk.start();

What Your First Dashboard Should Show

When your traces start flowing, build this dashboard first. It answers the questions you'll actually ask during an incident.

Health overview (top row):

Tool call success rate by tool name (last 24h)
P99 latency by tool (last 1h)
Active sessions with loop alerts

Volume signals (middle row):

Calls per tool per hour (trend line)
Top 10 sessions by call volume (detect outliers)
Error rate by error type

Debugging helpers (bottom row):

Recent failed traces (link to trace ID)
Slowest 10 tool calls (last 1h)
Authentication failures by client

Most of this is one or two queries in Grafana or Honeycomb once your spans have the right attributes. The hard part is deciding what to look at -- the dashboard above covers 80% of what matters.

Three Things That Will Bite You in the First Week

After shipping MCP observability across a few teams, these are the gotchas that consistently show up in the first week of production.

Baggage propagation breaks at async boundaries. OpenTelemetry baggage (where you store conversation ID, user ID, feature flags) relies on Node.js AsyncLocalStorage. If you're using a job queue, a worker pool, or any pattern that passes work across async contexts without explicit propagation, your downstream spans will lose the parent context. The fix: propagate trace context explicitly when crossing async boundaries.

import { context, propagation } from '@opentelemetry/api';
 
// When pushing to a queue or starting a worker
const carrier: Record<string, string> = {};
propagation.inject(context.active(), carrier);
 
// Store carrier in the job payload
await queue.push({ ...jobData, _otelContext: carrier });
 
// When the worker picks up the job:
const parentContext = propagation.extract(context.active(), job._otelContext);
await context.with(parentContext, async () => {
  // All spans created here inherit the parent trace
  await processJob(job);
});

SSE transport drops spans on reconnect. If you're still on SSE as your MCP transport (deprecated as of the 2025-03-26 spec), the agent reconnects if the connection drops. Each reconnect creates a new session ID. Without handling this, you'll see fragmented traces that look like separate conversations when they're actually one session. Track reconnect events explicitly and store the original session ID in a persistent store so you can stitch them together. Migrating to Streamable HTTP eliminates this class of problem entirely — sessions are decoupled from connections via the Mcp-Session-Id header.

Sampling drops your most important spans. Head-based sampling (deciding at the start of a trace whether to record it) is simple but wrong for AI agents. A conversation that starts normally can become your most important trace if the agent makes a bad tool call in turn 7. Use tail-based sampling: buffer all spans for a trace and make the sampling decision at the end, keeping any trace that contains errors, high latency, or flagged quality scores. Grafana Tempo and Honeycomb both support tail-based sampling. Datadog's adaptive sampling approximates it.

The Difference Between Running and Watching

An MCP server that's running isn't the same as one you can see. When something goes wrong at 2 AM, the difference between "running" and "watching" is how quickly you find the answer.

The setup in this guide takes about a day to instrument and another day to tune thresholds and build your initial dashboard. After that, you have a production system you can actually debug — which is a much better place to be than parsing raw logs and guessing what your agent saw.

If you'd rather not run the OTel collector yourself, a managed MCP runtime can ship traces, metrics, and loop detection out of the box — same OpenTelemetry conventions, no instrumentation work. Self-hosting on top of the official SDK is fine; you just write the wrappers above.