Continuous profiling

On this page 7

Continuous profiling is the practice of collecting performance profiles from production systems at all times, with low enough overhead that it can stay enabled permanently. In Observability for the Cloud, it shows which functions and lines of code consume CPU time, memory, locks or I/O, and how that consumption changes across deployments, hosts and time.

Metrics show that a service is slow, traces show which request and which span is slow, and logs record discrete events. Continuous profiling answers the next question: which code is responsible for the resource usage. For this reason it is often described as the fourth signal of observability, alongside metrics, logs and traces.

How continuous profiling collects data

Continuous profilers rely mainly on statistical sampling. Many times per second, often around 100 samples per second per CPU, the profiler interrupts execution and records the current call stack. Over minutes, the aggregated samples give an accurate picture of where time is spent, without instrumenting each function.

Collection happens through one of several mechanisms. eBPF-based agents on Linux read stacks from the kernel and work across languages without code changes, while language runtime profilers such as those built into Go, the Java virtual machine or .NET provide richer detail on allocations, locks and garbage collection. The agent periodically uploads compressed profiles, usually every 10 to 60 seconds, to a backend that stores them with labels such as service, version, region and pod.

The table below compares the main collection approaches.

MethodStrengthsLimitations
eBPF system-wide agentNo code changes, covers all processesSymbols needed for interpreted languages
Runtime profiler (SDK)Allocation, lock and heap detailRequires library in each service
Instrumentation profilingExact call countsHigh overhead, unsuitable for production
Hardware countersCache misses, branch mispredictionsPlatform-specific, harder to interpret
Profiling collection methods

Profile types and what they reveal

A single continuous profiler usually captures several profile types, each measuring a different resource. Choosing the right type matters because a service can be slow while consuming almost no CPU, for example when threads wait on a mutex or on the network.

Profile typeWhat it measuresTypical question answered
CPU timeOn-CPU stack samplesWhich code burns processor cycles
Wall-clock timeOn-CPU and off-CPU timeWhere requests wait
Heap allocationsBytes and objects allocatedWhat drives garbage collection
In-use memoryLive heap at a point in timeWhat causes memory growth or leaks
Lock contentionTime blocked on mutexesWhy threads stall under load
Common profile types

Results are usually displayed as flame graphs, where the width of each box represents the share of samples in that function. Differential flame graphs compare two periods or two versions and highlight which functions grew, which makes performance regressions visible within minutes of a deployment.

Continuous profiling compared with other signals

Continuous profiling complements the other telemetry types rather than replacing them. Its distinctive value is code-level attribution of resource cost across the entire fleet, which neither metrics nor distributed tracing provide on their own.

SignalGranularityPrimary use
MetricsAggregated numbers per serviceAlerting and trends
LogsIndividual eventsDebugging specific occurrences
TracesSpans within a requestLatency across services
ProfilesFunctions and lines of codeResource cost inside a process
Profiles and other observability signals

Modern platforms increasingly link profiles to traces by tagging samples with span identifiers, so an engineer can jump from a slow span to the exact code that ran during it. OpenTelemetry has added a profiling signal to its data model, which aims to standardize the format across vendors, although support is still maturing. Open-source backends include Grafana Pyroscope and Parca, and commercial APM suites such as Datadog offer integrated profilers.

Overhead, retention and deployment choices

Continuous profiling is viable only because sampling overhead is low, typically 1 to 5 percent of CPU depending on sampling rate, language and profile types enabled. Allocation and lock profiles cost more than CPU sampling, so many teams enable them selectively or at lower rates.

Storage is another design decision. Profiles are highly repetitive, and backends use stack deduplication and columnar compression to keep costs reasonable. Teams usually keep full-resolution data for days and downsampled aggregates for weeks or months to support long-term trend analysis.

SettingCommon choiceTrade-off
Sampling rateAbout 100 HzHigher rates add detail and overhead
Upload interval10-60 secondsShorter intervals raise network traffic
RetentionDays full, weeks aggregatedLonger history increases storage cost
Label cardinalityService, version, regionToo many labels inflate the index
Configuration choices and their trade-offs

Label design follows the same discipline as metric cardinality: adding a per-user or per-request label to profiles can multiply storage and slow queries without adding much insight.

Limitations and common mistakes

Continuous profiling is powerful but easy to misread. The most frequent problems stem from missing symbols, unrepresentative sampling and treating profile data as if it were exact.

  • Stripped binaries and JIT code produce unreadable stacks unless debug symbols or perf maps are available.
  • Short-lived processes, such as batch jobs or serverless functions, may finish before enough samples accumulate.
  • Sampling bias means rare but expensive code paths can be underrepresented in short windows.
  • CPU profiles alone miss latency caused by waiting, so wall-clock or off-CPU profiles are needed for I/O-bound services.
  • Optimizing without business context can waste effort on hot functions that do not affect user-facing latency or cost.

Profiles can also contain function names and file paths that reveal internal structure, so access control on the profiling backend deserves the same care as access to logs.

Understanding it simply

Continuous profiling is like a fitness tracker for software. It quietly watches a running program all day. Many times each second, it notes what the program is doing at that exact moment.

After a while, the notes show clearly where the effort goes. Engineers can see that one small piece of code uses half of the computer power. They can then fix that piece and save money or speed up the service. Because the watching is light, it can stay on all the time.

Applied in practice

A payments company runs about 300 containers for its checkout API. After a routine release, the cloud bill rises by roughly 18 percent, yet error rates and latency dashboards look normal. The team opens a differential flame graph comparing the new version with the previous one.

The graph shows that a JSON logging helper now serializes the full request body on every call, consuming about a fifth of total CPU time. The developers restore the previous log level, redeploy within an hour, and the profile returns to its earlier shape. The fleet scales back down by around 50 containers, and the team adds a profile comparison to its release checklist so that regressions are caught before costs grow.