Support memory profiling

[luweglarz]

Hello,

I am opening this issue because I thought it would be great to have memory profiling integrated to the opentelemetry-ebpf-profiler, and this is something I would like to tackle if it is decided that it is possible to do so.

Here is the strategy I imagine:

1. eBPF program on Uprobe Entry (malloc/calloc/realloc...): Captures the size parameter
2. eBPF program on Uretprobe Exit: Captures the returned memory address -> send alloc event
3. eBPF program on Uprobe on Free: Captures the address being freed -> send free event
4. User space needs to keep track of stack_traces holding > 0 memory alloc (not sure about how to make this part efficient)

For that I imagine that we could reuse the current eBPF architecture but with a TRACE_MEMORY origin, that would also send the memory alloc/free metadata and user space would do the correlation with trace_events and potential memory_events map.
Or creating a dedicated eBPF architecture for collecting stack_traces and memory metadata to be sent to user space when a memory alloc/free is triggered.

This is a naive intuition so far that would only capture virtual memory, but I think this is a beginning. so let me know what do you think !

[fabled]

While the technically this will likely work. The limitation is that the overhead to capture backtrace on all of the malloc calls can be significant. This probably is not problem in debugging like environment, but would probably cause issue if trying to run it in production or "always on". The magic in CPU profiling is that sampling happens often enough to be meaningful, but not often enough to have too high CPU overhead.

I wrote earlier a POC memory profiler for Java, where there was single Uprobe on the java internal function that gets called when the thread local allocation pool needs more memory. This had the effect to get Uprobes in a rate that was maintainable. The problem was that the Uprobe location needed manual disassembly as its not exported.

Also, you may want to further define what you want to achieve with "memory profiling". Is it just trying to get data on where most of the memory is allocated from? Or do envision to do some sort consistency checking (e.g. double free, freeing in allocated memory, etc.)?

If doing just CPU usage tracking, it might be an idea to have a preload library that hooks the functions, and provides simple wrappers for the memory allocators that track how much memory is allocated, and then expose an uprobe point when enough memory has been allocated. This would allow a more sampling like approach and potentially by runnable in production like environments.

OTOH, trying to do any consistency checks will require all calls to be sampled, and probably is not production feasible with ebpf approach.

Though, these are just my initial toughts, and doing some tests on how much the actual overhead is will give better insight on the feasibility.

[luweglarz]

What I mean by memory profiling is something similar to what we get when profiling Go applications, the memory that is currently in-use by the process, this is why I mention a sort of correlation between the allocation and free made by a specific stack_trace, but indeed as you mentioned the issue with that is that we cannot apply a sort of thresold as we have to capture each alloc/free calls for that.

Maybe another idea is to track only allocation from a certain size?

One first thing that could be done as test is to leverage the uprobe profiling already implemented to evaluate the overhead on memory symbols ?

[gnurizen]

Couple thoughts:

1. to be feasible and accurate and not biased memory profiling needs to be based on a sampling approach like Go and Jemalloc use where a sample is taken after N bytes of allocation volume, where N is something like 512kb.
2. for glibc a ld preloaded shim library like @fabled mentions probably makes the most sense although ideally it would be based on a usdt probe so it could be easily extended to other memory allocators
3. a completely different approach we're exploring is to just rely on the built in memory profilers and read their internal tables, http://github.com/parca-dev/oomprof implements this approach for Go and we have plans to add jemalloc. Although this approach relies on symbols being exposed and relies on the profilers internal state not changing.

@luweglarz which allocator did you have front of mind to target?

[luweglarz]

So I assume that every allocators would need to be attached to their specific usdt probe and handled differently? I thought we could rely on the classic ones like malloc to be agnostic but that sounds more complicated than that.

@gnurizen I had in mind ptmalloc, tcmalloc essentially for now, but wanted to start with malloc

[r1viollet]

+1 on the USDT probe approach. @nsavoire was trying this out with our allocator shim and it is really nice to get a cross language view of allocations with the OTel profiler.

This does require some kind of collaboration from the profiled process to expose probes in the sampled code paths (to avoid the overhead mentioned above). The shim injection mostly works with dynamically linked allocators. We would really need compile time support to get this working with statically linked allocators.

Keeping track of live allocation (leak tracking capabilities) also requires some help from the allocator, if you want to do it in an efficient way. Example: you want to have an extra flag that tells you if the object is being tracked (similar to what Go is doing).

[meastp]

hi @luweglarz, @fabled - disclaimer: I'm a user, and by no means a profiling expert, so this might be completely off the mark:

• I'd like to profile my current (C++) app, which does not have a specialized allocator.
• If the overhead is too high to use in production, would it be ok to have the functionality behind a flag/gate, so one is able to enable this in a test/debug environment?
• Have you considered bpftime? The ad says that userspace bpf overhead is significantly - 10x - lower. https://github.com/eunomia-bpf/bpftime
• I have used heaptrack before, perhaps memory profiling must be made as an external application (either with bpftime or LD_PRELOAD) which exports profiling data to an external collector instead of integrated into otel-ebpf-profiler?

[fabled]

* If the overhead is too high to use in production, would it be ok to have the functionality behind a flag/gate, so one is able to enable this in a test/debug environment?

Depending on the application and what it does, trying to inspect all allocations via kernel ebpf could still be too slow. For some programs it might work, for some others it might add too much overhead. Also depends how real-time sensitive the application is.

* Have you considered bpftime? The ad says that userspace bpf overhead is significantly - 10x - lower. https://github.com/eunomia-bpf/bpftime

At least I am aware of it. As we did not support uprobes earlier, this was unneeded. And even for uprobes, I am not sure how well it would do. The problem is that we use a lot of bpf_probe_read_user calls to interrogate the target process. And at times we may chase incorrect pointers, so the implementation must be safe to be given any random pointer. bpftime seems to have #define to control whether this is implemented in safe or unsafe manner. The safe manner will do syscalls for every single helper call. And that might result in tons of syscalls per backtrace generation. I suspect the end result might actually be slower than kernel bpf.

bpftime also breaks the other long time contract we have had. Completely transparent to target process. As injects code to the target process, it may also create stability issues and has definitely observable changes to the target process.

* I have used heaptrack before, perhaps memory profiling must be made as an external application (either with bpftime or LD_PRELOAD) which exports profiling data to an external collector instead of integrated into otel-ebpf-profiler?

I think the earlier suggest path of have LD_PRELOAD that overrides the memory management functions and creates a sampling spot for them sounds most viable. That is malloc is overridden in such a way that it calls default malloc but in addition executes uprobe code path every N th call or every N kB memory allocated. Since this will rate limit the ebpf trigger, running the kernel ebpf becomes feasible. There will be of course limitations for the data accuracy, and what can be determined from it. But this would at least be a basis for building a sampling memory profiler.

[meastp]

@fabled Thank you for the insight :)

I guess some form of support library must be made, then (using LD_PRELOAD).

If profiling all allocations isn't efficient enough through opentelemetry-ebpf-profiler, would one do better with a solution entirely in user space? Capturing all allocations and creating and pushing otlp profiling data directly to the collector?

Again, I'm not well versed in this domain, just mapping possibilities to have memory profiling which have been very useful to me (and having it available as part of otel observabilitiy would be truly awesome) :)