Skip to main content

Error Handling & Monitoring

This document describes error scenarios, logging, monitoring, and recovery strategies for the donation platform.

Error Categories​

1. User Input Errors​

Cause: Invalid or missing data from user

Examples:

  • Empty email field
  • Invalid email format
  • Amount less than minimum (€1.00)
  • Unsupported currency

Handling:

Requests are validated with standard Active Model validations on Donations::Request (email format, amount numericality, currency and period inclusion, €1.00 minimum after conversion to EUR). The controller checks valid? and renders the validation errors as JSON with a 422 status.

User Experience:

  • Frontend validation before API call
  • Server returns 422 Unprocessable Entity
  • Error messages displayed in UI
  • User can correct and retry

Response Example:

{
  "email": ["can't be blank", "is invalid"],
  "amount_cents": ["value must be at least €1.00"]
}

2. Payment Errors​

Cause: Payment processing failures

Card Declined​

Common Card Error Codes:

  • card_declined - General decline
  • insufficient_funds - Not enough balance
  • lost_card - Card reported lost
  • stolen_card - Card reported stolen
  • expired_card - Card expired
  • incorrect_cvc - Wrong CVC
  • incorrect_number - Invalid card number
  • processing_error - Temporary issue

User Experience:

  • Error message displayed with reason
  • User can try different card
  • Support email provided

Authentication Required (3D Secure)​

User Experience:

  • Redirected to bank's authentication page
  • Enter code or approve via app
  • Return to donation page
  • Success or failure message

Network Errors​

Handling:

  • Background jobs retry network/timeout errors with polynomially increasing waits (3 attempts at the Active Job layer)
  • Sidekiq then retries failed jobs with its standard exponential backoff (25 retries by default)
  • After all retries: job moves to the dead set, manual intervention required

3. Webhook Processing Errors​

Stripe webhooks are received by the stripe_event engine mounted at /stripe-webhook, which verifies the signature with the secret from STRIPE_WEBHOOK_SECRET_V2. Verified events are handed to StripeWebhookJob (queue: payments), which re-fetches the event from the Stripe API by ID and dispatches it to Donations::ProcessStripeEventJob.

Signature Verification Failed​

Causes:

  • Wrong signing secret configured
  • Request body modified (middleware/proxy)
  • Timestamp too old (>5 minutes)
  • Replay attack attempt

Impact:

  • Webhook rejected
  • Stripe will retry (up to 72 hours)
  • Manual investigation may be needed

Event Processing Failed​

Handling:

  • Job retried automatically (Sidekiq)
  • Exponential backoff between retries
  • After 25 failures: moved to dead queue
  • Failures reported to Rollbar (after the 5th Sidekiq retry, see below)

4. Fraud Detection​

A failed charge is treated as a fraud attempt when any of the following is true:

  1. Stripe fraud report: fraud_details.stripe_report is fraudulent
  2. Fraudulent decline code: the outcome reason is one of pickup_card, lost_card, fraudulent, stolen_card, merchant_blacklist
  3. Blocked by Stripe Radar: outcome type blocked with status not_sent_to_network, or outcome reason highest_risk_level
  4. Too many failures: more than 3 failed charges on the same payment intent within 30 minutes (tracked via Redis-backed rate limits)

Actions:

  • Payment intent expired immediately (Donations::ExpirePaymentIntentJob with reason fraudulent), preventing further attempts
  • A warning is logged
  • The "charge failed" email to the donor is suppressed

There is no separate staff alert for fraud: the failure still produces the regular Donations::Notification record (visible in Slack and in the admin panel).

5. Subscription Errors​

Failed Recurring Payment​

Stripe's Automatic Retry:

Recurring payment retries are handled by Stripe's dunning settings (Smart Retries), configured in the Stripe Dashboard. After the configured retries are exhausted, Stripe cancels the subscription automatically.

User Experience:

  • Email notification after each failure (unless classified as fraud)
  • Link to update payment method
  • Grace period before cancellation
  • Can update payment method to prevent cancellation

Subscription Creation Failed​

Handling:

  • Error message displayed to user
  • User can try different payment method
  • No subscription created in Stripe or database
  • Clean state, can retry from beginning

6. Database Errors​

Duplicate Transaction​

Scenario: Webhook replayed or delivered twice

Handling:

  • Unique index on transaction_id (donations_transactions table)
  • Insert fails silently
  • No duplicate transaction created
  • Idempotent webhook processing

Missing Donor​

Scenario: Donor exists but Stripe customer ID not yet saved

Handling:

  • Retrieve customer from Stripe API
  • Find donor by email
  • Update donor with customer ID
  • Continue processing normally

Logging Strategy​

Structured Logging​

Logging uses Semantic Logger (via the rails_semantic_logging gem), which adds structured tags — including the authenticated user — to every log line and enriches controller request logs with HTTP context for Datadog correlation (trace_id/span_id).

Log Output:

[2024-01-15T10:30:45.123Z] [INFO] [request_id=abc123] [ip=192.168.1.1] Processing stripe event evt_123 of type charge.succeeded

Log Storage​

Development:

  • log/development.log
  • Colorized console output
  • Detailed SQL queries

Production:

  • Stdout, captured by Heroku
  • Aggregated by the Logtail add-on and shipped to Datadog Logs
  • Searchable and filterable (Datadog log explorer, correlated with APM traces)

Sensitive Data Filtering​

Rails parameter filtering is configured for passw, secret, token, _key, crypt, salt, certificate, otp, ssn — preventing sensitive data from appearing in logs.

Error Tracking (Rollbar)​

  • Enabled in production only, using ROLLBAR_ACCESS_TOKEN (server-side) and ROLLBAR_CLIENT_TOKEN (browser JS)
  • All unhandled exceptions reported automatically with stack trace, scrubbed request parameters, and person tracking
  • Reporting is asynchronous (dedicated thread)
  • Sidekiq threshold: job failures are reported only after the 5th retry, avoiding noise from transient errors
  • Ignored exceptions: circuit-breaker open errors (Stoplight::Error::RedLight), ActiveRecord::RecordNotFound, AbstractController::ActionNotFound

Error grouping caveat: Rollbar/Error Tracking groups issues by exception class and raise site, not by message — HTTP client wrappers can bucket unrelated 4xx/5xx errors under one issue. Always read the most recent occurrence's message when investigating.

Monitoring & Metrics​

Application Performance Monitoring (APM)​

Datadog Integration (site: datadoghq.eu):

  • APM tracing enabled in production (DD_TRACE_ENABLED), including Heroku router request queuing as a dedicated heroku-router service
  • Runtime metrics enabled
  • Health check and asset requests filtered out of traces
  • Tracer logs routed through Semantic Logger

Metrics Tracked:

  • Request latency (p50, p95, p99)
  • Webhook processing time
  • Background job duration
  • Database query time
  • Stripe API response time

Custom Metrics​

Custom metrics are emitted through Metrics::MetricService, a thin wrapper around DogStatsD with the aleteia. prefix (gauges and counters with normalized tags). There are currently no donation-specific custom metrics — payment monitoring relies on APM traces, logs, and the Stripe Dashboard.

Health Checks​

The health-monitor-rails engine is mounted at the application root:

Endpoint: /check

Monitored providers:

  • Rails cache
  • Redis
  • Sidekiq (alerts when queue size exceeds 200)

Recovery Procedures​

Replaying Webhooks​

From Stripe Dashboard:

  1. Go to Developers → Webhooks
  2. Find the webhook endpoint
  3. Click on failed event
  4. Click "Resend"

Programmatically: since jobs are keyed by Stripe event ID, an event can be reprocessed from the Rails console with StripeWebhookJob.perform_later(event_id) (or Donations::ProcessStripeEventJob.perform_later(event_id) to skip dispatch).

Handling Failed Jobs​

The Sidekiq Web UI is mounted at /jobs (authenticated users only): retries and the dead set can be inspected and re-enqueued from there.

Database Rollback​

If bad data was imported, always:

  • Backup database before manual changes
  • Test in development/staging first
  • Document all manual interventions
  • Update monitoring after recovery

Alerting​

  • Errors: Rollbar notifies on new and reactivated error types
  • Donation activity: every donation event (donations, subscriptions, cancellations, failed charges) is posted to Slack via Donations::SlackNotificationJob
  • Infrastructure: Datadog monitors and the /check health endpoint cover services and queues

Best Practices​

Error Handling​

  1. Fail Fast: Validate early, fail explicitly
  2. Idempotency: All operations should be safely retryable
  3. Graceful Degradation: Partial feature failures shouldn't break entire system
  4. User-Friendly Messages: Don't expose technical details to users
  5. Context: Always log enough context to debug

Monitoring​

  1. Baseline Metrics: Establish normal values for all metrics
  2. Alert Fatigue: Too many alerts = all alerts ignored
  3. Actionable Alerts: Every alert should require an action
  4. Post-Mortem: Document and learn from incidents

Troubleshooting Guide​

Payment Not Processing​

Check:

  1. Is payment in Stripe Dashboard?
  2. Was webhook sent by Stripe? (Dashboard → Developers → Webhooks → endpoint attempts)
  3. Did webhook arrive at application? (search logs for the event ID)
  4. Was webhook processed successfully? (Sidekiq UI at /jobs: retries / dead set)
  5. Was transaction created in database? (Donations::Transaction.find_by(transaction_id: ...))

Donor Not Receiving Email​

Check:

  1. Was the transaction / notification record created?
  2. Was the failure classified as fraud? (fraud suppresses the donor email)
  3. Did the mailer job succeed? (Sidekiq UI, Rollbar)
  4. Did SendGrid accept the email? (SendGrid activity feed)
  5. Did the email bounce? (SendGrid suppressions)

Webhook Signature Verification Failing​

Check:

  1. Correct signing secret configured? (STRIPE_WEBHOOK_SECRET_V2 must match the endpoint's signing secret in the Stripe Dashboard)
  2. Request body being modified by a proxy/middleware?
  3. Using raw request body (not parsed)?