Watchtower

Services

The decision worth arguing about

The two costs that decide whether a self-hosted observability platform is cheap or a disaster are cardinality and tiering, and they behave in opposite ways to what people expect. Cardinality is not a storage problem. A compressed Prometheus sample is somewhere around one and a half to two bytes, so a million active series scraped every fifteen seconds produces roughly ten gigabytes a day of blocks — on S3 that is a rounding error. Cardinality is a memory problem: the ingest tier holds every active series' head chunk in RAM at roughly eight kilobytes a series, so a million series is about eight gigabytes per ingester replica, and you run three replicas for the replication factor. Double your label cardinality and you double the most expensive compute in the system while the storage bill barely moves. That is why Watchtower enforces per-tenant active-series limits at the distributor and reports series growth by team, and why the collector drops high-cardinality labels — pod name, request id, full URL path — before they ever reach an ingester. Tiering is where the second trap sits. It is tempting to lifecycle log chunks aggressively into Glacier Instant Retrieval, and for compliance retention beyond ninety days we do exactly that. But retrieval from Glacier tiers carries a per-gigabyte fetch charge, and the moment an engineer runs a four-month LogQL query during an incident they scan across the boundary and generate a bill nobody budgeted for. So the policy we deploy is deliberately conservative: thirty days in Standard, sixty more in Standard-Infrequent Access, and only then a Glacier tier that the query gateway refuses to read from without an explicit flag and a warning showing the estimated fetch volume. The other S3 detail worth naming is request cost rather than storage cost — a wide query fans out across thousands of small blocks and generates a lot of GET requests, which is why the memcached chunk cache is not optional and why the compactor's block sizes are tuned rather than left at defaults.

Services

The decision worth arguing about

Azure's blob tiers look like S3's but they bill differently in a way that quietly breaks the obvious lifecycle policy, and this is the Azure-specific decision worth understanding. Cool, Cold and Archive tiers each carry an early-deletion penalty: move a blob to Cool and delete it before thirty days and you are billed as if it had stayed thirty days; Cold is ninety days, Archive is one hundred and eighty. Observability backends delete constantly. The compactor's whole job is to merge small blocks into large ones and delete the originals, and log chunks get rewritten by retention enforcement. Put a naive lifecycle rule on the bucket that tiers everything after seven days and the compactor will spend the next month generating early-deletion charges on blocks it created and destroyed inside a fortnight. So on Azure we keep the entire compaction window — all levels, not just the newest — in Hot, and only apply tiering to blocks that the compactor has finished with and marked terminal, which the deployment tracks explicitly rather than inferring from age. The second Azure divergence is Event Hubs versus MSK: Event Hubs' Kafka surface is good enough for the ingest write-ahead log and cheaper to operate than a managed Kafka cluster, but it caps message size at one megabyte on the standard tier, so the collector's batch sizes are tuned down accordingly, which trades a little throughput efficiency for not having to run Kafka. And the reason we run this at all rather than pointing everything at Azure Monitor: Log Analytics bills per gigabyte ingested, which is the same meter shape as Datadog's, just with a Microsoft logo on it.