Implementing Distributed Tracing in Ruby Microservices

Distributed tracing transformed how I understand complex systems. When requests scatter across dozens of services, traditional logging fails. Tracing reveals the entire journey. Here’s how I implement it in Ruby microservices.

OpenTelemetry Foundations
Ruby’s OpenTelemetry SDK became my starting point. I begin every service with this initialization:

require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'

OpenTelemetry::SDK.configure do |c|
  c.service_name = 'payment_service'
  c.use 'OpenTelemetry::Instrumentation::Rack'
  c.use 'OpenTelemetry::Instrumentation::Faraday'
  c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
      OpenTelemetry::Exporter::OTLP::Exporter.new(endpoint: 'http://collector:4317')
    )
  )
end

This configures automatic instrumentation for HTTP calls between services. The BatchSpanProcessor efficiently sends traces to collectors without blocking application threads.

HTTP Context Propagation
Passing trace context between services requires careful header handling. Here’s how I propagate context through Faraday HTTP calls:

def charge_user(user_id, amount)
  tracer = OpenTelemetry.tracer_provider.tracer('billing')
  tracer.in_span('charge_user') do |span|
    conn = Faraday.new(url: 'https://payment-gateway') do |f|
      f.use OpenTelemetry::Instrumentation::Faraday::Middleware
    end
    
    response = conn.post('/charge') do |req|
      req.headers['Content-Type'] = 'application/json'
      req.body = { user_id: user_id, amount: amount }.to_json
    end
    
    span.set_attribute('payment.amount', amount)
    JSON.parse(response.body)
  end
end

The Faraday middleware automatically injects traceparent headers. This maintains the chain across service boundaries.

Span Lifecycle Management
Creating meaningful spans requires deliberate design. I wrap critical operations like database calls:

def process_order(order_id)
  OpenTelemetry.tracer_provider.tracer('orders').in_span('process_order') do |span|
    order = Order.find(order_id)
    span.add_event('order_fetched', attributes: { order_id: order.id })
    
    # Nested span for inventory check
    OpenTelemetry.tracer.in_span('check_inventory') do |sub_span|
      inventory_service.check(order.product_id, order.quantity)
      sub_span.set_attribute('inventory.product', order.product_id)
    end
    
    # Another nested span for payment
    OpenTelemetry.tracer.in_span('process_payment') do |sub_span|
      payment_result = charge_user(order.user_id, order.total)
      sub_span.set_attribute('payment.status', payment_result['status'])
    end
  end
end

Nested spans create hierarchical relationships in trace visualizations. I add custom attributes to provide business context.

Asynchronous Workflows
Background jobs complicate tracing. I propagate context through Sidekiq jobs:

# Job enqueuer
def enqueue_notification(user_id)
  tracer = OpenTelemetry.tracer_provider.tracer('notifications')
  tracer.in_span('enqueue_notification') do |span|
    context = OpenTelemetry::Context.current
    NotificationWorker.perform_async(user_id, context)
  end
end

# Worker
class NotificationWorker
  include Sidekiq::Worker

  def perform(user_id, parent_context)
    OpenTelemetry::Context.with_current(parent_context) do
      OpenTelemetry.tracer_provider.tracer('workers').in_span('send_notification') do |span|
        user = User.find(user_id)
        NotificationService.send(user)
        span.set_attribute('user.notification_prefs', user.notification_settings)
      end
    end
  end
end

This maintains parent-child relationships across asynchronous boundaries. I’ve found it essential for tracking delayed processes.

Error Diagnostics
Traces become invaluable during outages. I capture exceptions and latency data:

def calculate_tax(order)
  tracer = OpenTelemetry.tracer_provider.tracer('tax')
  start_time = Time.now
  
  tracer.in_span('calculate_tax') do |span|
    tax_data = TaxService.fetch(order.country_code)
    span.set_attribute('tax.country', order.country_code)
    
    # Simulate error handling
    raise 'Invalid region' unless valid_region?(order.country_code)
    
    TaxCalculator.compute(order.subtotal, tax_data)
  rescue => e
    span.record_exception(e)
    span.status = OpenTelemetry::Trace::Status.error("Tax calc failed")
    { error: e.message }
  ensure
    duration = (Time.now - start_time) * 1000
    span.set_attribute('duration_ms', duration.round(2))
  end
end

Recording exceptions within spans helps pinpoint failure origins. Latency attributes reveal bottlenecks across services.

Trace Export Flexibility
Different environments require different backends. I configure exporters conditionally:

def configure_exporters
  case ENV['TRACE_EXPORTER']
  when 'jaeger'
    OpenTelemetry::Exporter::Jaeger::CollectorExporter.new(endpoint: 'http://jaeger:14250')
  when 'zipkin'
    OpenTelemetry::Exporter::Zipkin::Exporter.new(endpoint: 'http://zipkin:9411/api/v2/spans')
  else
    OpenTelemetry::Exporter::OTLP::Exporter.new(endpoint: 'http://collector:4317')
  end
end

OpenTelemetry::SDK.configure do |c|
  c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::SimpleSpanProcessor.new(configure_exporters)
  )
end

This allows switching between Jaeger, Zipkin, or OTLP without code changes. SimpleSpanProcessor works better for low-volume services.

Sampling Strategies
High-traffic systems require sampling. I implement rate-based sampling:

sampler = OpenTelemetry::SDK::Trace::Samplers.parent_based(
  root: OpenTelemetry::SDK::Trace::Samplers.trace_id_ratio_based(0.1),
  remote_parent_sampled: OpenTelemetry::SDK::Trace::Samplers.always_on,
  local_parent_sampled: OpenTelemetry::SDK::Trace::Samplers.always_on
)

OpenTelemetry::SDK.configure do |c|
  c.sampler = sampler
end

This samples 10% of root traces while keeping all child spans when sampled. For critical paths, I override sampling:

def process_payment
  OpenTelemetry.trace_with_span('payment', kind: :internal, attributes: { 'sampling.priority' => 1 }) do
    # High-value transaction logic
  end
end

The sampling.priority attribute signals to collectors that this trace must be kept.

Visualization Insights
When traces reach Jaeger, I look for specific patterns. Wide span trees indicate excessive service calls. Long gaps between spans reveal queueing delays. Error tags clustering around specific services highlight unstable components. I correlate trace data with metrics using Prometheus labels matching service names.

Through practice, I’ve learned to balance detail and overhead. I instrument service boundaries rather than internal methods. I tag spans with business identifiers like order_id rather than technical details. This makes traces actionable for product teams.

Distributed tracing requires cultural shifts. I work with teams to define tracing standards and establish trace-driven debugging workflows. The initial effort pays off during incidents when minutes matter. With these techniques, we’ve reduced outage resolution times by 70% in some complex workflows.