Lens

Services

The decision worth arguing about

The most important decision in Lens is not to ship a query engine. The semantic layer compiles to SQL and pushes execution into the warehouse the customer already pays for, because a BI tool that maintains its own extracts becomes a second, stale copy of the business within a year. Where there is no warehouse, the default is Athena over Parquet in S3, and that is where the cost conversation has to be honest: Athena charges $5 per terabyte scanned, and a twelve-tile dashboard auto-refreshing every five minutes for forty concurrent users will scan tens of terabytes a month and can quietly cost more than every other line in this deployment combined. Four things keep that under control, and they are design constraints rather than settings. The compiler prunes to only the columns a tile actually uses, which is where columnar storage earns its keep. Partition projection is mandatory on any table above about 50 GB, so a date filter never triggers a partition scan. Every result is cached in Redis against a hash of the compiled SQL plus the partition watermark, and dashboards default to cached-until-invalidated rather than live, so a hundred people opening the same board at 09:00 scan once between them. And each tile displays the bytes it scanned, because the fastest way to stop an expensive dashboard is to show the analyst who built it what it costs. Past roughly 30 TB of scanning a month, the arithmetic flips and a small always-on Redshift Serverless workgroup or a DuckDB cache tier on Fargate is cheaper than per-terabyte billing; the compiler emits standard SQL either way, so that migration is a connection string.

Services

The decision worth arguing about

Azure offers the same choice as AWS and one that AWS does not, and picking between them is not really an engineering decision. Synapse serverless SQL bills per terabyte scanned, behaves like Athena, and rewards exactly the same discipline — column pruning, partitioning, caching. Microsoft Fabric bills the opposite way: you buy a fixed capacity in F-SKUs, reserve it for a year, and the price does not move regardless of how much you scan. The non-obvious part is what happens at the limit. Per-terabyte billing punishes a runaway dashboard with an invoice; fixed capacity punishes it by throttling, which means a single badly written tile degrades every query for everybody until the smoothing window clears, and nobody gets a bill to tell them why. So the choice is between a predictable experience with an unpredictable cost and a predictable cost with an unpredictable experience, and which one is correct depends on whether your finance team fears variance more than your analysts fear latency. We default to Synapse serverless below roughly 50 TB scanned a month and move to a reserved Fabric capacity above it, with the same compiler, the same caching and the same cost display on every tile. Microsoft does not render F-SKU rates on the public pricing page — they appear only through the calculator with a region selected — so we size that capacity per customer rather than publishing a number here that we cannot cite.