A short crash course on metrics: gauges, counters, and histograms
The metric type you pick is a promise about how the number behaves over time. Pick the wrong one and everything you build on top of it breaks silently. No error, no warning, just a graph that stops making sense until someone notices. Here is the small mental model that keeps you out of that.
Gauge: a value you can read right now
A gauge is a snapshot. It goes up and down freely, and a single reading means something on its own. Think of a speedometer or a thermometer. Both show you the value for right now, both rise and fall with conditions, and one glance tells you the current state. Active connections, queue depth, memory in use, temperature. You read the current value and that is your answer.
You never run rate() or increase() on a gauge, because the current number already is the thing you want.
The one trap is that a gauge only talks about the present moment. A speedometer tells you your speed as you look at it, not the distance you covered while you were not looking. If you only scrape it every five minutes, you have no idea what happened in the gaps. When that matters, max_over_time(), min_over_time() and avg_over_time() over the window get some of it back.
Counter: a number that only goes up
A counter counts events since the process started. Requests served, errors seen, messages processed. It only increases, and it resets to zero when the process restarts. That reset is the important part.
Picture a bouncer at a club with a hand clicker, counting people as they walk in. The clicker only goes up, and at the start of each night it goes back to zero. The raw number on the clicker is not very interesting. What the manager wants to know is how many people came in during the last hour, and you get that by subtracting the reading from an hour ago from the reading now. That subtraction, made safe against the reset, is what rate() and increase() do. Both are reset-aware: when the value drops, they assume a restart and correct for it, so you get real growth instead of a large negative jump.
Two rules save you here:
Never use a counter's raw value as if it meant something on its own. Always wrap it in
rate()orincrease().Apply
increase()per series before you sum across instances.sum(increase(x[5m]))is right.increase(sum(x)[5m])is wrong. Once you sum many instances into one line, that line drops every time any single instance restarts, andincrease()reads each drop as a fresh spike. A metric that should sit around a few hundred per minute can suddenly report tens of thousands from a routine deploy.
One more thing: rate() needs at least two samples in its window to return anything, so keep the window at a few times your scrape interval. [1m] with a 30s scrape is already on the edge.
And if a number is really a counter, export it as a counter. Dressing it up as a gauge to get past some check does not change what it is. It just moves the breakage onto whoever queries it later.
Histogram: distributions, not averages
Sometimes one number cannot answer the question. Take request latency. If you only track the average, a healthy endpoint and a badly broken one can look the same, because the average hides the fact that one request in twenty takes ten seconds. You need the shape of the values, not a single summary.
A histogram gives you that shape by dropping each observation into a bucket. You define the boundaries up front, say 10ms, 50ms, 100ms, 500ms, 1s, and every request lands in one. The buckets are cumulative: the 100ms bucket counts every request that finished in 100ms or less, the 500ms bucket counts everything at 500ms or less, and so on. From those totals you can read what fraction of traffic came in under any boundary you picked.
Under the hood each bucket is a counter, plus two more counters alongside them: a _sum of every observed value and a _count of how many there were. Because every part is a counter, the same discipline applies. Rate first, then compute, never read the raw totals.
Two useful things come out of this cheaply:
histogram_quantile()over the rated buckets gives you percentiles. p50 is the latency most requests beat, p95 is the tail, p99 is the slow edge.Rated
_sumdivided by rated_countgives you a real average over the window, instead of a lifetime average that barely moves.
If p50 is 40ms but p99 is 3s, most people are fine and a real slice of traffic is suffering. A single average would have erased that.
Two things to keep in mind. Percentiles from a histogram are only as good as your buckets: histogram_quantile() interpolates inside the bucket the quantile falls in, so if p99 lands in a bucket that runs from 1s to 10s, the number is a guess. Put boundaries where you actually care about precision. And you cannot average or sum percentiles across instances. To get a fleet-wide p99 you aggregate the buckets first, sum by (le) (rate(...)), and then run histogram_quantile() on that.
Summary: the other distribution type
A summary looks like a histogram but computes the quantiles in the client, at observation time. You get _sum, _count and a few precomputed quantiles like p50, p90, p99 straight out, with no bucket tuning. The downsides are real though: the quantiles are over a sliding window baked into the client, they cost more CPU, and you cannot aggregate them across instances at all. Reach for a histogram unless you have a specific reason not to.
A note on labels
Every distinct combination of label values is a separate time series that has to be stored and queried. Putting a user ID, a request path with IDs in it, or anything else high-cardinality into a label is the fastest way to melt your metrics backend. Keep labels to a small, bounded set of values.
A note on naming
Two conventions worth following. Suffix counters with _total, so http_requests_total, not http_requests. It signals the type at a glance and some tooling keys off it. And put the base unit in the name: seconds, not milliseconds, and bytes, not megabytes. request_duration_seconds and memory_usage_bytes. You lose nothing by storing 0.04 instead of 40, and every dashboard and alert stops guessing which unit it is looking at.
Seeing it in a real service
Concepts stick better with something running in front of you. Here is a tiny Spring Boot service with a single endpoint, POST /orders, that exercises all three types at once. The full project is in blog/code/metrics-demo.
POST /orders sleeps for a randomized processing time, takes a slow path roughly one time in fifteen, and rejects roughly one order in ten. That is enough to make each metric type say something interesting.
Wiring it up
Two dependencies. spring-boot-starter-actuator adds the /actuator endpoints, and micrometer-registry-prometheus teaches one of them to speak the Prometheus exposition format.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
The /actuator/prometheus endpoint is not exposed by default, so turn it on:
management:
endpoints:
web:
exposure:
include: health,info,prometheus
The instrumentation
All three meters are created once, in the constructor, from the injected MeterRegistry.
@RestController
public class OrderController {
private final Counter accepted;
private final Counter rejected;
private final Timer processingTime;
private final AtomicInteger inFlight = new AtomicInteger(0);
public OrderController(MeterRegistry registry) {
// counter: one metric name, one series per outcome label
this.accepted = Counter.builder("demo_orders_total")
.tag("outcome", "accepted").register(registry);
this.rejected = Counter.builder("demo_orders_total")
.tag("outcome", "rejected").register(registry);
// gauge: reads straight off the AtomicInteger on every scrape
Gauge.builder("demo_orders_in_flight", inFlight, AtomicInteger::get)
.register(registry);
// histogram: only the bucket boundaries we care about, plus the implicit +Inf
this.processingTime = Timer.builder("demo_order_processing_seconds")
.serviceLevelObjectives(
Duration.ofMillis(25), Duration.ofMillis(50),
Duration.ofMillis(100), Duration.ofMillis(250),
Duration.ofMillis(500), Duration.ofSeconds(1))
.register(registry);
}
@PostMapping("/orders")
public ResponseEntity<String> createOrder() throws InterruptedException {
inFlight.incrementAndGet();
try {
return processingTime.recordCallable(this::handle);
} catch (Exception e) {
throw new IllegalStateException(e);
} finally {
inFlight.decrementAndGet();
}
}
// handle() sleeps a random time and increments accepted or rejected
}
The naming from the previous section is already in play: demo_orders_total gets the _total suffix because it is a counter, and demo_order_processing_seconds is in seconds, not milliseconds.
What comes out
curl localhost:8080/actuator/prometheus, after some traffic:
# TYPE demo_orders_total counter
demo_orders_total{application="metrics-demo",outcome="accepted"} 34865.0
demo_orders_total{application="metrics-demo",outcome="rejected"} 3824.0
# TYPE demo_orders_in_flight gauge
demo_orders_in_flight{application="metrics-demo"} 5.0
# TYPE demo_order_processing_seconds histogram
demo_order_processing_seconds_bucket{application="metrics-demo",le="0.025"} 7965
demo_order_processing_seconds_bucket{application="metrics-demo",le="0.05"} 25776
demo_order_processing_seconds_bucket{application="metrics-demo",le="0.1"} 36060
demo_order_processing_seconds_bucket{application="metrics-demo",le="0.25"} 36060
demo_order_processing_seconds_bucket{application="metrics-demo",le="0.5"} 36261
demo_order_processing_seconds_bucket{application="metrics-demo",le="1.0"} 37900
demo_order_processing_seconds_bucket{application="metrics-demo",le="+Inf"} 38689
demo_order_processing_seconds_count{application="metrics-demo"} 38689
demo_order_processing_seconds_sum{application="metrics-demo"} 3594.364214676
Everything from the theory is visible here. The counter has one series per outcome and both values only climb. The gauge is a single number. The histogram is a stack of _bucket counters, one per le boundary, cumulative (each count already includes every lower bucket), plus _sum and _count. The le="0.1" and le="0.25" buckets hold the same count because nothing in this workload finishes between 100ms and 250ms: orders are either quick or they take the slow path.
A small Prometheus scrapes this endpoint every 5 seconds. The graphs below are its expression browser.
The counter: raw versus rated
The raw counter just climbs. Two lines, one per outcome. The height is meaningless, it only reflects how long the process has been up, and the drop to zero at 10:18 is a restart. The slope is the whole story.
sum by (outcome) (rate(demo_orders_total[1m])) turns the slope into orders per second, around 48/s accepted and 6/s rejected. Two things to notice. The restart that sent the raw value to zero is barely a blip here, because rate() sees the drop, assumes a reset, and corrects for it. And sum by (outcome) is doing the "rate first, then aggregate" step: rate() runs per series, and only then do instances collapse into one line per outcome.
The gauge
demo_orders_in_flight, plotted directly. No rate(), no increase(). It bounces between 0 and roughly the number of concurrent clients, and every point on the line is a real reading of how many orders were mid-flight at that instant.
The histogram
Two histogram_quantile() calls over the rated buckets, p50 and p95 on one graph. The median order finishes in about 35ms, but p95 sits near 640ms, because one order in fifteen takes the slow path and that tail is exactly what a percentile is built to expose. The average alone, about 90ms, would have blended the two into one number that describes nobody.
Look at where p95 lands: up in the wide 0.5s to 1s bucket. That means the value is histogram_quantile() interpolating across a 500ms gap, so read it as "somewhere in that range", not a measurement. If that range mattered, the fix is more bucket boundaries there.
sum(rate(..._sum[5m])) / sum(rate(..._count[5m])) gives a true average over the window from the same two counters, instead of a lifetime average that barely moves.
The short version
Before choosing a type, ask one question. Does this number stand on its own right now, or is it a count of things that happened?
Stands alone: gauge. Read it directly.
A count: counter. Always rate or increase it, and always per series before summing.
The shape of many values: histogram. Its parts are counters too, so the same rules apply.
Almost every metric bug is one of these three promises being quietly broken.
