Modern applications run in dynamic environments, often across microservices and containerised platforms, leading to complex architecture evolution. Performance regressions are more likely to be introduced with each new release, which segues naturally into the importance of profiling to measure resource usage and maintain application health.
We started relying on Go’s built-in pprof tool for performance analysis of applications but realised soon that it was just a snapshot; it failed us when analysing performance trends over time was the requirement. This is where continuous profiling comes in. By collecting profiles at regular intervals, teams can analyse historical data, detect regressions early, and make data-driven optimisations in production environments.
pprof stands for performance profiler and is Go’s built-in performance analysis tool designed to help developers understand where their application spends time and how it uses resources.
Ever felt confused trying to figure out why your application is running slow? Or where CPU time is consumed, and are there any memory leaks? Or which goroutines are blocking each other?
Go’s pprof captures different runtime profiles to answer these questions:
To profile a Go program, import the net/http/pprof package that exposes HTTP endpoints like /debug/pprof/profile , /heap and /goroutine, and install handlers under the /debug/pprof URL to download live profiles. You can read more about the available endpoints and usage in the official Go documentation. When a profile is requested, Go’s runtime collects the program’s execution data. This data is stored in a binary format (protobuf) that the pprof tool can later read and visualise.
Example:
To analyse the CPU performance, run:
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
This collects a 30-second CPU profile (by default), then launches an interactive CLI where you can run sub-commands.
However, the main limitation of using pprof is that it shows us the performance output of a particular point in time for the application, and it becomes difficult to keep track of performance bottlenecks over time or compare history across releases; that is where Parca comes into the picture.
Parca is a continuous profiling tool that collects, stores, and makes profiles available to be queried over time. Due to the nature of sampling profiling, it is possible that some parts of an execution are missed; therefore, continuous profiling attempts to gather data continuously at a low overhead so that with enough data it is statistically significant. It scrapes pprof endpoints for application-level data and uses eBPF for zero-instrumentation CPU profiling. The roadmap (related work here) suggests that eBPF-based memory and network profiling will be available in future releases, which will eliminate the need for application instrumentation for these profile types. But if you want to check other profiles beyond CPU, you will need to add profiling configuration in the Parca configuration file and ensure the target application exposes these endpoints like this.
Check how different profiles are scraped through Parca's Ingestion Documentation. Designed for cloud-native environments; Parca integrates seamlessly with Kubernetes through an agent-based architecture, making it ideal for microservices. Parca consists of two components: Parca Server that stores profiling data and enables querying over time, and Parca Agent, an eBPF-based system profiler that captures data across the entire system.
Learn more in the official Parca documentation.
Figure 1: Continuous profiling workflow with Parca in Kubernetes
The illustration shows continuous profiling with Parca on Kubernetes. Go applications expose pprof endpoints that Parca Agent discovers via pod annotations. The agent captures profile data (including CPU, heap, and goroutines) and sends it to the Parca Server, which stores it for periodical time-series analysis. Parca UI queries this data to display flame graphs and time series comparisons to the user, providing end-to-end visibility into bottlenecks and historical tuning insights.
Follow this quick-start guide to see how continuous profiling actually works!
Create a namespace
kubectl create namespace parca
Install Parca Server (Check the latest Parca Server release here)
kubectl apply -f https://github.com/parca-dev/parca/releases/download/v0.24.1/kubernetes-manifest.yaml
Install Parca Agent (Check the latest Parca Agent release here)
kubectl apply -f https://github.com/parca-dev/parca-agent/releases/download/v0.41.0/kubernetes-manifest.yamlRefer to a sample go application here.
Don’t forget to import the /net/http/pprof package and run a HTTP server to expose the /debug/pprof/ endpoints. The underscore import (import_ "net/http/pprof") ensures that the package’s init() function runs, which automatically registers the profiling handlers even though the package isn’t directly referenced in your code.
By convention, port 6060 is often used for profiling, but you can configure any port as well.
Wrap your Go app inside a container image to run it inside a Kubernetes cluster. This image will be used for profiling along with Parca. We’ve used Docker Hub in this demo, but you can push to any container registry (Vayu Cloud’s Container Registry, Docker Hub, GCP Artifact Registry, AWS ECR, GitHub Container Registry, etc.) depending on your environment.
You can test pprof by port-forwarding the Pod/Service and accessing /debug/pprof/ in the browser or using the go tool pprof in CLI.
Example:
For CPU profile:
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
For Heap profile:
go tool pprof http://localhost:6060/debug/pprof/heap
For different memory profiles:
• --inuse_space: The size of current live objects in bytes (heap memory in use).
• --inuse_objects: The number of objects currently live (not garbage collected).
• --alloc_space: The total number of bytes ever allocated since the program started.
• --alloc_objects: The total number of objects ever allocated since the program started.
Parca supports two types of scraping:
1. Static Scraping: You manually list the targets in the parca configmap.
Figure 2: Static scraping with fixed target
While static scraping is suitable for isolated testing or single-instance applications, it lacks the scalability required for dynamic Kubernetes environments and incurs significant manual overhead, which leads us to the concept of Dynamic scraping.
2. Dynamic Scraping (via annotations/service discovery)
Instead of listing every endpoint, Parca relies on Kubernetes service discovery.
In the parca configmap, the user must specify kubernetes_sd_configs.role as either pod or service depending on where the annotations are being added. This will tell Parca which type of Kubernetes resource to query and watch through the API. The first relabel rule keeps only pods with parca.dev/scrape: "true". The second rule reads the pod’s IP and the value from annotation parca.dev/port and combines them as <ip>:<port>.
Figure 3: Parca configuration file with role as pod
Here’s what happens:
Profiling data is exposed by your Go application on port 6060 via net/http/pprof . Parca automatically finds the Pod via the Kubernetes API and starts scraping from /debug/pprof/profile . Refer to this deployment for example.
Parca can also do service-based discovery. When enabled, Parca scrapes via the Service’s FQDN instead of chasing changing Pod IPs. This is useful when your app has multiple replicas behind a Service, Parca can hit the Service endpoint instead of each Pod individually.
To access the Parca UI, expose the Parca Server service using your Ingress Controller (example Istio), and open it in your browser to see something like this:
Figure 4: Parca UI displaying profile types and flamegraph
Parca UI displays flamegraphs, icicle graphs, and differential views for different profiles, helping you analyse and optimise application performance.
Note: Make sure your application shows up in the Parca UI under the Targets tab. A healthy status here confirms that Parca is successfully scraping profiling data from your application. Parca UI displays CPU profiles by default through its eBPF-based continuous profiling engine, without requiring application restarts or code modifications, provided your application has the correct annotations.
While setting up a prototype is a necessary first step, the true value of continuous profiling is best seen in a production environment. Let’s look at a specific performance bottleneck we identified and resolved using this exact setup.
We have two Grafana automations running in Vayu Cloud's Kubernetes-as-a-Service, we started noticing a continuous increase in their memory consumption over the time, we were puzzled as to why this would happen as the automations do very similar things like adding a Grafana dashboard to a folder and setting the Grafana RBAC.
The Grafana dashboard showed us something was wrong, i.e., memory usage was increasing, and pods were occasionally getting OOMKilled, but it stopped at the symptom. We could see memory going up, but not which part of the code was responsible for it.
Figure 5: Grafana Dashboard showing high memory usage trend
This is where Parca made a difference. We were running two Knative functions in production: grafana-rbac and grafana-dashboard.
Both showed memory pressure, but Parca helped us figure out which functions from the code were responsible for this. Once profiling was enabled, patterns became immediately visible.
Note: Each Parca graph includes two lines: blue for grafana-dashboard and yellow for grafana-rbac. The analysis in this section focuses on the grafana-dashboard function to clearly highlight improvements.
Parca revealed through the Memory In-Use Bytes graph that memory increased during reconcile activity and didn’t consistently return to its earlier baseline.
Figure 6: Parca graph showing spikes in memory in-use bytes
Figure 7: Parca’s flamegraph highlighting Configmapcontroller as the highest memory contributor
The Memory Allocated Bytes Total graph showed steep, repeated ramps, each corresponding to reconcile activity, which suggested repeated expensive operations.
Figure 8: Parca graph of total allocated memory bytes over time
Figure 9: Parca flamegraph illustrating overall memory allocation across components
The Goroutine Delta graph showed spikes during reconcile cycles, which also showed that the system was repeatedly creating background work rather than reusing existing resources.
Figure 10: Parca graph depicting goroutines spikes during reconciles
Figure 11: Parca flamegraph depicting multiple goroutines running parallelly
By selecting a point in time, Parca generated a heap flame graph that broke down memory usage by function. Across multiple runs, the same call stack consistently appeared as a major contributor.
Once we saw this stack, the issue became clear in the code; every reconcile created a new Kubernetes ConfigMap informer. Each informer listed every ConfigMap in the cluster, deserialised them all into Go objects, and started a background goroutine.
After identifying the issue, we made a targeted change by removing the ConfigMap controller from the reconcile path and validating the impact, which resulted in an immediate difference in the profiling data. The dominant stack related to the ConfigMap informer initialisation was significantly no longer a top contributor.
Figure 12: Grafana dashboard showing improved and stabilised memory usage
Figure 13: Parca graph showing reduced memory in-use bytes for the Grafana dashboard Knative function
Figure 14: Parca flamegraph showing optimised memory distribution with no dominant memory-consuming component
While the behaviour improved, memory usage patterns still indicated expensive operations happening during reconciliation, which taught us an important lesson: profiling is not a one-time fix; it's an iterative process.
Parca helped us validate both the improvements and the remaining inefficiencies. It was a design pattern problem: expensive Kubernetes operations (informers, API calls, decoding) were being initialised inside request handlers instead of being reused.
While this blog focuses on Parca, it's important to briefly understand how it compares with Pyroscope, another leading open-source continuous profiling tool.
Parca employs a distributed architecture heavily inspired by Prometheus, with separate components: the Parca Agent (eBPF-based profiler) and Parca Server (storage and query engine). The Prometheus-inspired design means teams already familiar with Prometheus will have a quick learning curve around pod discovery, querying, targets etc. Parca also has its own UI available (apart from the Grafana integration) which can be used to do more granular profiling. Pyroscope (now part of Grafana Labs) follows a push-based model where applications actively send profiling data via client SDKs or language agents. It offers an all-in-one binary that handles both profiling and storage, with broader language support including Python, Ruby, Java, and Go. While it supports eBPF profiling through Grafana Alloy integration, it also provides language-specific SDKs for applications, making it more versatile for heterogeneous environments.
Parca's distributed architecture requires more initial setup but provides better scalability for large deployments. Teams familiar with Prometheus will find configuration patterns similar. The eBPF-first approach means minimal application changes but requires kernel support. Pyroscope's single-binary deployment simplifies initial setup, especially for smaller teams, but the SDK-based approach requires application instrumentation.
Both tools share the same goal, continuous visibility into performance over time, but differ in architecture and how they operate. The choice often comes down to your existing infrastructure, team expertise, and specific profiling requirements.
Parca enables continuous profiling in Kubernetes by scraping applications profiling data and displaying it in UI. Flamegraphs, time comparisons, and historical analysis make it easy to pinpoint performance issues and optimise code at peak performance.
Start your cloud-native journey with Vayu Cloud's Kubernetes-as-a-service, where you can deploy and scale applications seamlessly in a secure and production-ready environment. Explore observability tools like Parca and Pyroscope individually to understand how continuous profiling enhances performance visibility and empowers you to build more efficient, production-ready applications. As you grow, you can further level up by integrating these with other observability add-ons for a complete insight into your system’s behaviour.