Observability

Distributed tracing

Tracing a single message as it moves through the API, outbox, router, and carrier connector is essential for modern operations. The ub3r platform assigns a unique trace ID at the very edge of the system and propagates it through every internal service. This allows us to visualize the entire journey of a message in a single view, identifying exactly where delays or failures occurred. Without distributed tracing, debugging a multi-service CPaaS is like looking for a needle in a haystack.

We also include metadata in our traces, such as the carrier ID and the destination country, to help identify patterns in failures. If we see a spike in latency, we can quickly filter our traces to see if it is limited to a specific provider or region. This high-cardinality data is invaluable for performing root-cause analysis on complex production issues. Investing in tracing infrastructure pays for itself the first time you solve a "heisenbug" in minutes instead of days.

Core metric signals

We focus on the "four golden signals" of monitoring: latency, traffic, errors, and saturation, tailored for the communications domain. For ub3r, this means tracking the time from API acceptance to carrier submission, and the final delivery status of every message. We monitor our error rates not just as a global percentage, but broken down by carrier, tenant, and error code. This granularity allows us to spot localized outages that might otherwise be hidden in a global average.

Saturation is monitored by looking at our queue depths and the age of the oldest message in our outbox. A growing queue isn't always a problem, but a queue that isn't draining at all is a major incident. We use these metrics to drive our auto-scaling logic, ensuring we have enough workers to handle traffic bursts. Metrics provide the high-level view that tells us what is happening, while traces tell us why.

// Example of a custom metric for message latency
metrics.histogram("message_delivery_seconds", duration, {
  carrier: "provider_a",
  status: "delivered",
  region: "us-east-1"
});

Alerting strategies

Effective alerting is about signaling that a human needs to take action, not just that a threshold was crossed. We avoid "flappy" alerts by using moving averages and window-based logic to filter out transient noise. Our alerts are categorized by severity, with critical issues waking up an engineer and minor anomalies simply filing a ticket for the next business day. Every alert in the ub3r system is linked to a runbook that provides clear steps for resolution.

We also implement "negative alerts" to detect the absence of expected events, such as a carrier that has stopped sending delivery reports entirely. This is often a more reliable indicator of an outage than an increase in explicit error codes. By monitoring the "pulse" of the system, we can detect silent failures before our customers do. A proactive approach to observability is what allows us to maintain our 99.99% uptime commitment.