What observability to wire up in week one, and what to skip

The Tumbuhku API had distributed tracing on its second day and no dashboard for months. That ordering was on purpose, and it is the argument of this piece. On a small Go service, tracing pays for itself immediately, metrics dashboards mostly do not, and the expensive mistake is instrumenting everything while alerting on nothing.

The service in question is a Go and Echo backend over PostgreSQL, handling child growth records, milestones, and immunization data for a Next.js user app and a Next.js backoffice. Low traffic. Small team. High consequence per record. That combination changes the answer, and We will say where it does.

Trace first, because you cannot add context after the fact

A log line tells you what a developer decided to write down in advance. A trace tells you what the request actually did: which handler, which queries, in what order, how long each took, and where the time went that you did not account for. The second one cannot be reconstructed later. If a request was slow last Tuesday and you were not tracing, that information is gone.

Day one instrumentation is three integrations and roughly an hour of work: the OpenTelemetry middleware for Echo, a tracing wrapper on the pgx pool, and instrumented transport on any outbound HTTP client. That gives you a span per request, a child span per query, and a child span per external call. You have not written a single manual span yet and you can already answer most latency questions.

Span naming is where people quietly ruin this. A span name must be low cardinality. Take it from the Echo route pattern, so a request becomes a span named for the templated path with the parameter placeholder intact, never the resolved path with a real identifier in it. Put the identifier in an attribute if you need it, or preferably do not. A span name containing a UUID turns your backend's grouping into a list of unique strings, and every aggregate view becomes useless. Database spans get the operation and the table, with the statement text on db.statement.

Attributes, and the ones that must never leave the process

Useful attributes are the ones you would filter on: route, status code, tenant, rows affected, which feature flag was on, which reference standard version was resolved. Those are cheap and they answer real questions.

Then there is the category that must never be attached. In Tumbuhku the records are children's health data, so the list is not abstract: no name, no birth date, no weight, no height, no z-score, no immunization status, no photo URL, no phone number, no address, no free text a parent typed. Not on a span, not on a Sentry breadcrumb, not in a log line, not in an error message.

Birth date is the one that slips through. It does not feel identifying on its own, it is genuinely useful for debugging because every growth calculation depends on age in days, and somebody will add it to a span while chasing an off-by-one in an age boundary. Combined with a district and a sex it is close to identifying, and it is health-adjacent by context. The same reasoning applies to the measurement value itself: a weight in grams is a medical observation about a specific child.

Two controls make this practical rather than aspirational. First, the attribute list is an allowlist enforced in a span processor, not a denylist of things to strip. A denylist fails silently every time someone adds a field. Second, error reporting runs with default PII collection turned off and a send hook that drops request bodies, strips the Authorization and Cookie headers, and truncates anything unrecognised. If you need to correlate a report to a record, attach a hash or an opaque identifier that only resolves inside your own database.

Structured logs with a request ID beat a metrics stack you never open

Structured JSON logging through the standard library's slog, with one middleware that pulls trace_id and span_id out of the span context and attaches them to every line, plus a request ID and the route. That is the whole logging design. It costs an afternoon and it converts log search into a debugger: paste a trace ID, get the exact sequence of events for one request across handler, database, and background work.

What we did not build in week one: a Prometheus deployment, Grafana, custom dashboards, recording rules, an SLO document. For a service handling a few requests per second, a p99 latency computed from a histogram is mostly sampling noise, and a dashboard nobody has a reason to open is a screen that gets ignored during the one incident where it mattered.

There are four numbers worth having early: request rate, error rate, p95 latency, and database connection pool saturation. Your hosting platform and your database provider hand you most of them without any code. The pool one is the one people miss and it is usually the first thing to break, because a slow query plus a fixed pool size is how a small service turns a mild regression into total unavailability. Set MaxConns deliberately and watch waiting acquires.

Sampling when traffic is low

Sample everything. At a few requests per second, one hundred percent head sampling is affordable, and the alternative is discovering that the trace you needed was the one that got dropped. Head sampling decides before the request finishes, which means it cannot preferentially keep errors: it drops your 500s at exactly the rate you configured.

Reduce volume at the source instead of through the sampler. Filter health check and readiness paths out of instrumentation entirely so they never become spans. That removes the majority of the noise on a low traffic service without touching the sample rate for anything that matters.

The point to revisit this is when trace cost becomes visible on an invoice or when a single endpoint starts producing enough volume that your backend rejects it. At that point the answer is tail sampling in a collector, where the decision is made after the trace completes and can keep every error and every slow request. Introducing that machinery before you need it means running a collector, and a collector you have to operate, for a service that produced eleven traces an hour.

The error you page on and the error you read on Monday

These are different categories and treating them the same is what makes alerting worthless.

Page on: the service cannot serve. Boot failed because a migration did not apply. The connection pool is saturated and requests are queueing. Sustained error rate above a threshold for several minutes rather than a single spike. The weekly Friday deploy failing its health check. Every one of those has an obvious next step and somebody would want to be woken up.

Read on Monday: a 400 because a client sent a malformed date. A context deadline on a third party that retried and succeeded. One 500 in a handler that has served ten thousand clean requests. These belong in an issue tracker with a weekly triage, and they are genuinely worth reading, because that malformed date is usually a client bug you can fix at the contract level.

One configuration detail decides whether the second category stays readable: group errors by type and code location, not by message. Messages contain identifiers, and identifier-bearing messages shatter one recurring problem into four hundred distinct issues. Set an explicit fingerprint when the default grouping is doing that.

The rule We would defend: an alert must name an action. If the runbook step is look at it when you get a chance, it is not an alert, it is a report, and routing it to a pager teaches the team to mute the channel. Once the channel is muted the alerting system is worse than not having one, because you now believe you are covered.

When all of this is overkill

For an internal tool with a handful of users, a scheduled job, or a prototype with a demo date, tracing infrastructure is not the right spend. Structured logs with a request ID, and error reporting. That is the whole list, and it is enough to debug almost anything at that scale.

Running your own collector on day one is also overkill for most small services. Export directly to your backend and add a collector when you have a concrete reason: routing to two destinations, central redaction, or tail sampling. Infrastructure added in advance of a reason is infrastructure you maintain without a payoff.

The week one list, then: trace the boundaries, name spans after routes, allowlist your attributes, log with a trace ID, and configure two alerts that each name an action. Everything else can wait for a question you actually have.