Reliability · SRE

SLOs from scratch: picking your first SLI for a network service

Your first SLO should measure what your consumers experience, be based on current performance — not aspiration — and ship with a budget policy. Miss any of the three and you have built a dashboard, not an SLO.

Primer·platform & network engineers·

Scope: Article 17 of the reliability series. The canon is the Google SRE Book and SRE Workbook; the worked example is a real hybrid-DNS path (Route 53 Resolver inbound endpoints + on-prem conditional forwarders). Burn-rate math is handed to a later article on alerting; team-SLO politics to a later article. Numbers are arithmetic, shown.

You have been told to “define SLOs for the DNS (Domain Name System) service,” you open a blank doc, and the first question stops you cold: what number, measured how? You have a hundred threshold alarms that tell you when a box is unhappy and not one of them tells you whether the service is reliable. This is where the first number comes from — and for a network service, the hard part is not the target. It is deciding what even counts as an event.

TL;DR — the first SLO, in five moves

  • Keep the four terms surgically separate. SLI = good events / valid events (a ratio); SLO = that ratio’s target over a window; error budget = 1 − SLO, made spendable; SLA = the looser, penalty-bearing external contract. Blur them and every later decision is muddy.
  • For a network service, the hard part is defining the event. Web services have requests; you have flows, packets, and queries — pick one unit and one measurement point, as close to the consumer as you can afford.
  • Set the number from measurement and dependency math — never by vibes. Start just under what you already achieve; your SLO cannot exceed the composite of what you depend on. 99.99% = 4.32 min/month; one BGP (Border Gateway Protocol) flap ends it.
  • An SLO without a budget policy is decoration. Decide before the incident what happens when the budget is spent — change freeze, reliability work jumps the queue.
  • Alert on burn rate, not the SLI threshold. Threshold alarms are the exact problem SLOs replace; burn rate pages you for what actually threatens the budget.
Part 1Why, and the four words

The problem SLOs solve

It’s 3 a.m. and the pager reads resolver-2 CPU 87%. You ack it — and DNS answered every query the whole time. Two weeks earlier, resolution silently failed for eight minutes mid-deploy and nothing paged, because no one had defined what “DNS is up” even means. That is the gap. Threshold alarms answer “is this component unhappy?” — CPU over 80%, an endpoint down. Nobody has answered the only question that matters to the people who depend on you: “is the service reliable?” That gap costs you twice. You get alert fatigue, because every component pages independently, and you get no shared definition of “good,” so every team argues reliability by anecdote. An SLO is that missing shared definition, written as a single number both sides agree to. Everything below is how you arrive at that number honestly.

The SLO loop — everything, in one picturefollow 1 → 6 — each arrow says what that step does1measure it2vs. a target3= 1 − SLO4watch the burn5when it is spent6freeze + fixSLAlooser contract, penaltiesYour serviceRoute 53 ResolverSLIgood events ÷ valid eventsSLOtarget 99.9%, 30-day windowError budget1 − SLO = 43.2 min / 30 daysAlertingfast = page · slow = ticketBudget policyspent → freeze + reliability first
  1. Your service, measured — the SLI is computed from events you capture here.
  2. The SLI: good events ÷ valid events — the reliability number itself.
  3. The SLO: a target for that number over a window (and the looser SLA beside it).
  4. The error budget: 1 − SLO, made spendable, draining as you fail.
  5. Alerting watches how fast the budget burns — fast-burn pages, slow-burn tickets.
  6. The budget policy spends the consequence (a freeze + prioritized reliability work), and the fix closes the loop.
Figure 1 · The whole SLO loop in one picture — the rest of this article lights up one box at a time.

The four words, defined once

Most writing blurs these, and the blur is where teams get lost. Keep them surgically separate. An SLI is a measured ratio; an SLO is a target for that ratio over a window; an error budget is what is left over, 1 − SLO, turned into something you can spend; an SLA is the external contract with penalties, deliberately looser than the SLO so you react long before you owe anyone a refund. The SLO is your early-warning line; the SLA is the legal line.

Reliability · DNS query success · rolling 30 dayslive99.4%99.5%99.6%99.7%99.8%99.9%100.0%window start−25d−20d−15d−10d−5dnowrolling 30-day window — it slides right; a failure older than 30 days ages out and the budget recoversSLA 99.5% — the legal line (refunds below)SLO 99.9% — the target / early-warning lineerror budget = the green band = 1 − SLO = 0.1% ≈ 43.2 min / 30 daysyourmarginan incident — the SLI drops, spending budgetSLI — the measured good ÷ valid ratio
Figure 2 · The four terms on one dashboard. The SLI is the measured line (good ÷ valid); the SLO is a target line you set (99.9%); the SLA is a looser line with penalties (99.5%); the error budget is the room between them (1 − SLO). The incident spends budget but never reaches the SLA — you react at the SLO, long before you owe a refund.
The four words are four different objectsSLI · SLO · error budget · SLA — keep them separate1measure it2vs. a target3= 1 − SLO4watch the burn5when it is spent6freeze + fixSLAlooser contract, penaltiesYour serviceRoute 53 ResolverSLIgood events ÷ valid eventsSLOtarget 99.9%, 30-day windowError budget1 − SLO = 43.2 min / 30 daysAlertingfast = page · slow = ticketBudget policyspent → freeze + reliability first
Figure 3 · The four words are four different objects — an SLI (the ratio), the SLO (its target over a window), the error budget (1 − SLO), and the looser SLA. Keep them separate.
TermWhat it isExample
SLI
Service Level Indicator		A ratio: good events / valid events. A number between 0 and 1, measured — an indicator.good queries ÷ valid queries — e.g. over 30 days (≈100 queries/min): 4,318,000 good ÷ 4,320,000 valid = 99.95%
SLO
Service Level Objective		A target for the SLI over a window. Target + window.99.9% of valid DNS queries answered good, rolling 30 days
Error budget1 − SLO, made spendable — the allowed amount of “not good,” a quantity you run down.99.9% → a 0.1% budget → 4,320 bad queries (≈ 43.2 min) allowed per 30 days
SLA
Service Level Agreement		A contract with a customer, carrying penalties. Looser than the SLO on purpose.99.5% or credits — you breach the SLO long before this
Part 2The event, and a worked example

The hard part: what is an “event” for a network?

Web services have an obvious unit — the request — so good requests / valid requests writes itself. Network and infrastructure services do not: your units are flows, packets, and queries, and you have to choose one. Three families of SLI are on offer:

  • Availability — probe or health success: successful probes / total probes.
  • Latency — a percentile (p99, p99.9), never an average — averages hide the tail users actually feel; one 10-second query vanishes in a mean.
  • Correctness, loss, or throughput — packet loss, answered-correctly rate, or throughput against a target.
It starts with where you measurethe SLI is computed from events captured at the service1measure it2vs. a target3= 1 − SLO4watch the burn5when it is spent6freeze + fixSLAlooser contract, penaltiesYour serviceRoute 53 ResolverSLIgood events ÷ valid eventsSLOtarget 99.9%, 30-day windowError budget1 − SLO = 43.2 min / 30 daysAlertingfast = page · slow = ticketBudget policyspent → freeze + reliability first
Figure 4 · Everything starts at the service: the SLI is computed from events you capture there — so the first decision is where and what to measure.

For our DNS service those aren’t abstract: availability is the canary’s success rate, latency is the p99 of resolution time, and correctness is the non-SERVFAIL rate. You pick one to start.

Then the load-bearing choice: where do you measure? You have three options, and they trade closeness-to-the-user against cost. Figure 5 puts them on the actual path.

Where do you measure? The DNS resolution path and its SLI capture points Consumer app(on-prem) On-prem forwarderconditional forward Direct Connectthe path Route 53 Resolverinbound · 2 AZs AZ-a AZ-b Private zone(the answer) 1synthetic canary 2passive flow 3query logs Principle: measure as close to the consumer as you can afford. 1closest to user · samples, an extra moving part 2real traffic · coarse, usually no RCODE 3cheap, has RCODE · but server-side, misses the path The pick: ② query logs + ① a consumer canary
  1. Synthetic canary (consumer-side) — you generate traffic from where the consumer sits. Closest to real experience and works with no live traffic, but it samples and is one more thing to run.
  2. Passive flow data (the path) — NetFlow, VPC Flow Logs, or a tap on the Direct Connect segment. Real traffic, no injection, but coarse and usually blind to whether the answer was good.
  3. Service-side query logs (the endpoint) — the Resolver’s own per-query records. Cheap, already emitted, and carry the RCODE — but measured at the server, so they miss the network path to the consumer.
Figure 5 · You must choose where to measure. Server-side logs plus a consumer-side canary is the common affordable compromise — the logs are nearly free and carry semantics, the canary catches the path they cannot see.

One more thing this diagram assumes: for an infrastructure service, “the consumer” is other teams, not end users. That changes who you negotiate the number with — a topic a later article takes up and this one does not.

Measurement pointWhat it seesCost
① Synthetic canaryThe consumer’s real experience, path includedExtra moving part; samples, not every event
② Passive flowReal traffic on the wireCoarse; no app semantics, usually no RCODE
③ Service-side logsEvery query at the endpoint, with RCODENearly free, already on; blind to the path to you

Worked example: your first SLI for hybrid DNS

Here is the real shape. Apps on-prem resolve names in AWS private zones, and vice versa. On-prem DNS servers hold conditional forwarders that send AWS-domain queries over Direct Connect to Route 53 Resolver inbound endpoints — Elastic Network Interfaces (ENIs) living in two Availability Zones (AZs) — while outbound endpoints and rules carry AWS→on-prem queries the other way. Three SLIs are candidates: (a) query success rate, (b) resolution latency p99, (c) endpoint reachability.

Pick query success rate as the first one, for two reasons. It is closest to user pain: a failed resolution means the app cannot connect at all — a hard outage, not a slow one. And it is the cheapest to measure: Resolver query logs already emit a per-query result, so you add just one canary resolving a known name from the consumer side to catch the path failures the server cannot see. Now make the event definition explicit — this is the whole SLI, and vagueness here poisons every number downstream.

The event definition — DNS query success
• valid = queries that reach the Resolver endpoint. A query dropped before the endpoint is not counted bad here — the canary is what catches that gap, and you name the seam honestly rather than hide it.
• good = answered with a non-SERVFAIL response (SERVFAIL = DNS response code (RCODE) 2 = resolver failure) within 100 ms (illustrative) — correct-ish and timely.
• SLI = good queries / valid queries, measured over 30 days, from Resolver query logs plus the consumer canary.

Concretely, here is what lands in each bucket over a handful of real queries:

A query…Which bucket
A? app.aws.internal reaches the endpoint, answered NOERROR 10.20.3.4 in 8 msvalid + good ✓
same query, answered SERVFAIL (the upstream resolver was down)valid, not good — spends budget
same query, answered NOERROR but after 240 msvalid, not good — correct, but too slow
a query dropped on the Direct Connect path, never reaching the endpointnot valid — excluded here; the canary catches this gap
A? example.com — a public name this service doesn’t servenot valid — not this SLI’s job
That is the SLI. The rest is just the number, the budget, and the alarm — each one shorter than the decision you just made.
Part 3The number, the budget, the alarm

Setting the target number

Three inputs decide the number, in order — and none of them is a hunch. First, current measured performance: watch the SLI for two weeks — say you find you answer 99.93% of queries good — then set the SLO just under that, at 99.9%, reachable on day one. An SLO you are already violating teaches everyone to ignore it; tighten later. Second, dependency math: your SLO cannot exceed the composite of what you depend on. Availability of a serial path is roughly the product of its parts, so if the Direct Connect path runs 99.97% and on-prem DNS 99.97%, their composite is about 99.94% — that is your ceiling, so 99.9% is honest but 99.95% is already off the table. Redundancy (the two AZs) raises a component’s number; serial dependencies multiply it down. Third, the cost of another nine: each nine runs roughly 10× the last in redundancy and ops — going from 99.9% to 99.99% buys back just 39 bad minutes a month for ten times the effort — so buy only the nine the consumer actually needs.

Setting the numberthe SLO target and window — and the looser SLA beside it1measure it2vs. a target3= 1 − SLO4watch the burn5when it is spent6freeze + fixSLAlooser contract, penaltiesYour serviceRoute 53 ResolverSLIgood events ÷ valid eventsSLOtarget 99.9%, 30-day windowError budget1 − SLO = 43.2 min / 30 daysAlertingfast = page · slow = ticketBudget policyspent → freeze + reliability first
Figure 6 · The SLO is the target and the window; the SLA sits beside it as the looser, penalty-bearing promise.

What is forbidden is picking 99.99% because it sounds professional. That is 4.32 minutes of budget in 30 days; a single BGP flap blows it. Pick from measurement and dependency, not vibes.

InputWhat it tells you
Current measured performanceWhat you already hit (e.g. 99.93% over two weeks) — set the SLO just under it, achievable on day one
Dependency math (serial ≈ product)The hard cap: you cannot exceed the composite of what you depend on (Direct Connect 99.97% × on-prem DNS 99.97% ≈ a 99.94% ceiling)
Cost per nine (≈ 10× each)Whether the next nine is worth buying — only the consumer’s real need justifies it

Error budget, shown — and the policy everyone skips

The budget is just arithmetic, and seeing it makes the target real. 30 days = 43,200 minutes, and the budget is (1 − SLO) × 43,200. So a bad deployment that SERVFAILs for 20 minutes spends about half a 99.9% monthly budget in one shot; one BGP flap dropping the DX path for 5 minutes is roughly 12% of it. Budgets are small; incidents are expensive — that is the point.

The error budget is 1 − SLO, made spendable99.9% over 30 days = 43.2 minutes to spend1measure it2vs. a target3= 1 − SLO4watch the burn5when it is spent6freeze + fixSLAlooser contract, penaltiesYour serviceRoute 53 ResolverSLIgood events ÷ valid eventsSLOtarget 99.9%, 30-day windowError budget1 − SLO = 43.2 min / 30 daysAlertingfast = page · slow = ticketBudget policyspent → freeze + reliability first
Figure 7 · The error budget is 1 − SLO, made spendable — 99.9% over 30 days is 43.2 minutes to spend.
SLO (30-day)Error budget / 30 days
99%432 min (7.2 h)
99.9%43.2 min
99.95%21.6 min
99.99%4.32 min
Minutes are a translation, not the meter. The budget is really 0.1% of valid events — here, 4,320 bad queries in 30 days. “43.2 minutes” is only what that many failures equals if the service is 100% failing at its normal ~100 queries/min. Fail at half that rate and the same budget lasts ~86 minutes; break at 3am when no queries arrive and you spend nothing. You burn budget by answering events badly, not by the clock ticking.

The number is the easy half. The half that separates an SLO from a chart nobody acts on is the budget policy: what happens, decided in advance, when the budget is spent.

Budget policy — write it before the incident. When the budget for the window is exhausted: a change freeze on the service (no non-critical deploys) until it recovers, and reliability work jumps the queue ahead of features. No policy, no SLO — just a decoration.

Alerting on the SLO

Do not alert on the SLI threshold. “SERVFAIL rate over 1% right now” recreates the exact problem SLOs replace — it is the component-unhappy alarm again, firing on blips that never threaten the budget. Alert on burn rate instead: how fast you are spending the budget. A fast burn — say 2% of the monthly budget in an hour — should page; a slow burn — 10% over a day — should open a ticket. Figure 9 shows one incident tripping each at a different moment.

Alert on how fast the budget burnsfast-burn pages, slow-burn tickets1measure it2vs. a target3= 1 − SLO4watch the burn5when it is spent6freeze + fixSLAlooser contract, penaltiesYour serviceRoute 53 ResolverSLIgood events ÷ valid eventsSLOtarget 99.9%, 30-day windowError budget1 − SLO = 43.2 min / 30 daysAlertingfast = page · slow = ticketBudget policyspent → freeze + reliability first
Figure 8 · You alert on how fast the budget burns, not on the raw SLI — fast-burn pages, slow-burn tickets.
One incident, two alarms — fast-burn pages first, slow-burn tickets later0%25%50%75%100%day 51015202530error budget remainingdays in the rolling windowon-pace budget line (sustainable)budget remainingone incident① fast-burn (short window) → PAGE — minutes in② slow-burn (long window) → ticket — hours later
  1. The budget sits near full until the incident, then the sharp dip drains it — the steeper the dip, the faster the burn.
  2. Falling below the shallow slow-burn line first (a long-window check, higher up) opens a ticket — you are now burning faster than the whole-month pace.
  3. Falling further, below the steep fast-burn line (a short-window check, lower down), sends a page — the same incident, now unmistakably urgent.
Figure 9 · One acute incident trips the short-window fast-burn alarm first (a page, minutes in); the long-window slow-burn alarm catches the same drain later (a ticket). Choosing the exact windows and multipliers is multi-window burn-rate alerting — the subject of a later article.
Part 4Ship it, and where it bites

How to implement it on AWS

The definitions are done — here is where the events actually come from. Three moves, collect → compute → watch, and each one is a managed service you already pay for. The pipeline below is the whole thing.

How to implement it on AWS — collect, compute, watchResolver query logs → a CloudWatch SLI metric → a burn-rate alarm → a pageRoute 53 Resolverthe hybrid-DNS serviceyou are measuringQuery logging → CW Logsevery query: RCODE +latency, per resolutionSLI metric (CW)metric math: good ÷ valid= the SLI, over 30 daysBurn-rate alarmfast-burn = pageslow-burn = ticketAmazon SNSpages theon-call ownerCW Synthetics canaryprobes from the consumerside — catches the pathCloudWatch dashboard — the contractSLI · budget burned · this window12345
  1. Collect — turn on Route 53 Resolver query logging. It writes every query — with its response code (RCODE) and latency — to CloudWatch Logs. That is your raw event stream, already carrying the good/bad label.
  2. Compute the SLI — a CloudWatch metric (metric math over a metric filter) counts “non-SERVFAIL within 100 ms” over valid queries: that ratio is the SLI. A CloudWatch Synthetics canary resolving a known name from the consumer side feeds the same metric, covering the path the server-side logs cannot see.
  3. Watch the burn — a CloudWatch alarm on the burn rate, not the raw SERVFAIL rate: fast-burn pages, slow-burn opens a ticket.
  4. Page — the alarm publishes to Amazon SNS, which pages the on-call owner.
  5. Publish — a CloudWatch dashboard shows the SLI, the budget burned, and the window. That dashboard is the contract you hand to consumer teams.
Figure 10 · Collect with Resolver query logs, compute the SLI as a CloudWatch metric (plus a Synthetics canary for the path), then alarm on burn rate and page through SNS — every box is managed AWS, nothing to run yourself.

Two honest seams: the query logs are measured at the endpoint, so the canary is what catches a broken path to the consumer; and the metric is only as fresh as log delivery, so lean on the canary — not the logs — for sub-minute paging.

Operationalizing

An SLO is a living agreement, so treat it like one. Revisit it on a cadence — monthly or quarterly. Publish the dashboard as a contract with the consumer teams: the SLI, the target, the budget burned, this window. Tighten only when you consistently beat the target with budget to spare; loosen only with the consumer’s agreement, and never to hide chronic pain. Keep the whole agreement on one page so nobody has to reverse-engineer it later.

The policy closes the loopa spent budget triggers a freeze and prioritized reliability work1measure it2vs. a target3= 1 − SLO4watch the burn5when it is spent6freeze + fixSLAlooser contract, penaltiesYour serviceRoute 53 ResolverSLIgood events ÷ valid eventsSLOtarget 99.9%, 30-day windowError budget1 − SLO = 43.2 min / 30 daysAlertingfast = page · slow = ticketBudget policyspent → freeze + reliability first
Figure 11 · The budget policy turns a spent budget into action (a change freeze + prioritized reliability work), and the fix closes the loop back to the service.
One-page SLO doc — the whole contract
• Service — hybrid DNS resolution (Route 53 Resolver inbound)
• SLI spec — valid = queries reaching the endpoint; good = non-SERVFAIL within 100 ms; measured at Resolver query logs + consumer canary
• Target — 99.9% (from measurement, not aspiration)
• Window — rolling 30 days
• Budget policy — on exhaustion: change freeze + reliability work jumps the queue
• Owners — the platform/network team, with the named consumer teams

Where it bites — failure modes

Every one of these is easy to fall into and each quietly turns the SLO back into the dashboard it was supposed to replace.

FailureFix
Measuring what is easy to scrapeMeasure what the consumer actually feels — forward from experience, not backward from convenience
A 100% targetLeave a budget; with zero budget you can never safely deploy
Five SLIs when one would doStart with the single SLI closest to pain; add more only if it earns its keep
Averages instead of percentilesUse p99/p99.9 — the mean hides the tail users feel
Treating the SLO as the SLAThe SLO is the tighter internal target; the SLA is the looser external contract
Deriving the SLI from a metric you already haveDefine the event from user experience first, then find how to measure it

References

  1. Google — Site Reliability Engineering, ch. “Service Level Objectives” (SLI / SLO / SLA definitions, error budgets). sre.google/sre-book/service-level-objectives
  2. Google — The Site Reliability Workbook, ch. “Implementing SLOs” (choosing an SLI, the event model, measurement, target from current performance). sre.google/workbook/implementing-slos
  3. Google — The Site Reliability Workbook, ch. “Alerting on SLOs” (burn rate, fast-burn/slow-burn, multi-window multi-burn-rate). sre.google/workbook/alerting-on-slos
  4. AWS — Amazon Route 53 Resolver: inbound and outbound endpoints, forwarding rules (the hybrid-DNS path). docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver.html
  5. AWS — Resolver query logging (per-query records used as the service-side SLI source). docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-query-logs.html
  6. IETF — RFC 1035, Domain Names — Implementation and Specification (DNS RCODEs; RCODE 2 = server failure, SERVFAIL). rfc-editor.org/rfc/rfc1035