Continuous profiling is the practice of collecting performance profiles from production systems at all times, with low enough overhead that it can stay enabled permanently. In Observability for the Cloud, it shows which functions and lines of code consume CPU time, memory, locks or I/O, and how that consumption changes across deployments, hosts and time.
Metrics show that a service is slow, traces show which request and which span is slow, and logs record discrete events. Continuous profiling answers the next question: which code is responsible for the resource usage. For this reason it is often described as the fourth signal of observability, alongside metrics, logs and traces.
How continuous profiling collects data
Continuous profilers rely mainly on statistical sampling. Many times per second, often around 100 samples per second per CPU, the profiler interrupts execution and records the current call stack. Over minutes, the aggregated samples give an accurate picture of where time is spent, without instrumenting each function.
Collection happens through one of several mechanisms. eBPF-based agents on Linux read stacks from the kernel and work across languages without code changes, while language runtime profilers such as those built into Go, the Java virtual machine or .NET provide richer detail on allocations, locks and garbage collection. The agent periodically uploads compressed profiles, usually every 10 to 60 seconds, to a backend that stores them with labels such as service, version, region and pod.
The table below compares the main collection approaches.
| Method | Strengths | Limitations |
|---|---|---|
| eBPF system-wide agent | No code changes, covers all processes | Symbols needed for interpreted languages |
| Runtime profiler (SDK) | Allocation, lock and heap detail | Requires library in each service |
| Instrumentation profiling | Exact call counts | High overhead, unsuitable for production |
| Hardware counters | Cache misses, branch mispredictions | Platform-specific, harder to interpret |
Profile types and what they reveal
A single continuous profiler usually captures several profile types, each measuring a different resource. Choosing the right type matters because a service can be slow while consuming almost no CPU, for example when threads wait on a mutex or on the network.
| Profile type | What it measures | Typical question answered |
|---|---|---|
| CPU time | On-CPU stack samples | Which code burns processor cycles |
| Wall-clock time | On-CPU and off-CPU time | Where requests wait |
| Heap allocations | Bytes and objects allocated | What drives garbage collection |
| In-use memory | Live heap at a point in time | What causes memory growth or leaks |
| Lock contention | Time blocked on mutexes | Why threads stall under load |
Results are usually displayed as flame graphs, where the width of each box represents the share of samples in that function. Differential flame graphs compare two periods or two versions and highlight which functions grew, which makes performance regressions visible within minutes of a deployment.
Continuous profiling compared with other signals
Continuous profiling complements the other telemetry types rather than replacing them. Its distinctive value is code-level attribution of resource cost across the entire fleet, which neither metrics nor distributed tracing provide on their own.
| Signal | Granularity | Primary use |
|---|---|---|
| Metrics | Aggregated numbers per service | Alerting and trends |
| Logs | Individual events | Debugging specific occurrences |
| Traces | Spans within a request | Latency across services |
| Profiles | Functions and lines of code | Resource cost inside a process |
Modern platforms increasingly link profiles to traces by tagging samples with span identifiers, so an engineer can jump from a slow span to the exact code that ran during it. OpenTelemetry has added a profiling signal to its data model, which aims to standardize the format across vendors, although support is still maturing. Open-source backends include Grafana Pyroscope and Parca, and commercial APM suites such as Datadog offer integrated profilers.
Overhead, retention and deployment choices
Continuous profiling is viable only because sampling overhead is low, typically 1 to 5 percent of CPU depending on sampling rate, language and profile types enabled. Allocation and lock profiles cost more than CPU sampling, so many teams enable them selectively or at lower rates.
Storage is another design decision. Profiles are highly repetitive, and backends use stack deduplication and columnar compression to keep costs reasonable. Teams usually keep full-resolution data for days and downsampled aggregates for weeks or months to support long-term trend analysis.
| Setting | Common choice | Trade-off |
|---|---|---|
| Sampling rate | About 100 Hz | Higher rates add detail and overhead |
| Upload interval | 10-60 seconds | Shorter intervals raise network traffic |
| Retention | Days full, weeks aggregated | Longer history increases storage cost |
| Label cardinality | Service, version, region | Too many labels inflate the index |
Label design follows the same discipline as metric cardinality: adding a per-user or per-request label to profiles can multiply storage and slow queries without adding much insight.
Limitations and common mistakes
Continuous profiling is powerful but easy to misread. The most frequent problems stem from missing symbols, unrepresentative sampling and treating profile data as if it were exact.
- Stripped binaries and JIT code produce unreadable stacks unless debug symbols or perf maps are available.
- Short-lived processes, such as batch jobs or serverless functions, may finish before enough samples accumulate.
- Sampling bias means rare but expensive code paths can be underrepresented in short windows.
- CPU profiles alone miss latency caused by waiting, so wall-clock or off-CPU profiles are needed for I/O-bound services.
- Optimizing without business context can waste effort on hot functions that do not affect user-facing latency or cost.
Profiles can also contain function names and file paths that reveal internal structure, so access control on the profiling backend deserves the same care as access to logs.
Understanding it simply
Continuous profiling is like a fitness tracker for software. It quietly watches a running program all day. Many times each second, it notes what the program is doing at that exact moment.
After a while, the notes show clearly where the effort goes. Engineers can see that one small piece of code uses half of the computer power. They can then fix that piece and save money or speed up the service. Because the watching is light, it can stay on all the time.
Applied in practice
A payments company runs about 300 containers for its checkout API. After a routine release, the cloud bill rises by roughly 18 percent, yet error rates and latency dashboards look normal. The team opens a differential flame graph comparing the new version with the previous one.
The graph shows that a JSON logging helper now serializes the full request body on every call, consuming about a fifth of total CPU time. The developers restore the previous log level, redeploy within an hour, and the profile returns to its earlier shape. The fleet scales back down by around 50 containers, and the team adds a profile comparison to its release checklist so that regressions are caught before costs grow.