pkg/proc: process spawned event #4171

firelizzard18 · 2025-10-02T03:00:41Z

Adds a ProcessSpawned event to lay the groundwork for multi-process DAP debugging. Updates #3947.

firelizzard18 · 2025-10-02T03:12:56Z

This is groundwork for implementing the StartDebugging DAP reverse request, to allow multiprocess DAP debugging, once #4078 is merged. But also, I want to be able to intercept the Process DAP event in an extension, detect that the process is NodeJS, and create a NodeJS debugger; one of my projects involves a Go program that spawns a NodeJS child. But I'm running into an issue.

(dlv) target follow-exec -on
(dlv) c
Warning: no debug info found, some functionality will be missing such as stack traces and variable evaluation.
Spawned new process 'node main.ts.js' (117495)
Stopped at: 0x0
=>   1: no source available
Command failed: could not read debug info (decoding dwarf section info at offset 0x0: too short) and could not read go symbol table (could not read section .gopclntab)

A node process obviously doesn't have .gopclntab. However, it appears that the process is still being stopped. I could exclude it by using a regex for follow-exec, but that defeats the purpose - I want to send a process event via DAP that my extension can intercept. @aarzilli Is it feasible to work around this? If a process does not have .gopclntab (and/or other Go-specific sections) is it reasonable to say "This definitely is not a Go binary so we should let it run and not attempt to debug it"?

aarzilli · 2025-10-02T12:23:14Z

What's a nodejs debugger? How does it work?

pkg/proc/target_group.go

service/dap/server.go

firelizzard18 · 2025-10-02T17:59:53Z

By "nodejs debugger" I mean "a debugger capable of debugging a JavaScript program executed with NodeJS". As in, what delve does for Go programs, but for a NodeJS program. This is the process flow I'm envisioning:

I tell vscode to launch a Go program.
vscode-go tells dlv to execute the program in dap-mode with follow-exec enabled.
dlv builds and executes the program.
The Go process fork-execs a NodeJS process.
dlv follows the fork-exec and emits a process spawned event.
dlv detects that the new process is not a Go process and lets it continue.
My extension intercepts the process spawned event (via DAP via a debug adapter tracker).
My extension interrogates the process, identifies it as a NodeJS process, and instructs vscode to attach a NodeJS debugger to it.
Now I am able to debug both the parent Go process and the child NodeJS process.

It is possible to do this manually - to wait for the NodeJS process to be spawned and manually launch a NodeJS debug session - but it is clumsy and error prone and the process I describe above should be a much better UX. To be crystal clear, delve plays no role in setting up the other debugger besides sending the process spawned event. All I want is for delve to send a notification without keeping the child stopped.

aarzilli · 2025-10-03T12:18:33Z

By "nodejs debugger" I mean "a debugger capable of debugging a JavaScript program executed with NodeJS". As in, what delve does for Go programs, but for a NodeJS program

Yes, that I understood. I was more wondering how those work. If they work like delve, by using ptrace (and ptrace-like things), or if it's more like JVM debuggers that talk to the VM. If it is the former there is no way to hand off a debugged process to another debugger, I investigated this possibility when I was implementing the multiprocess debugging functions of delve (being able to have a delve process per debugged process would have made things much simpler). Also you can't have two debuggers attached to the same process and there is also no way to detach from a process while leaving it stopped.
Or, at least, I couldn't figure out how to do it.

aarzilli

There are a lot of test failures, many due to the addTarget bug, others, I think, due to the fact that the DAP tests are now receiving two process events for the first process. If you search for newEvent("process") in service/dap/server.go you'll see that we were already sending one.

pkg/proc/target_group.go

firelizzard18 · 2025-10-03T17:51:34Z

If you search for newEvent("process") in service/dap/server.go you'll see that we were already sending one.

Oops, I should have started with that. I see that's in onLaunchRequest so I guess it only happens for the first/parent process? My intent was to emit an event for new child processes, I wasn't thinking about the parent process. I'll work on getting those tests passing.

I was more wondering how those work. If they work like delve, by using ptrace (and ptrace-like things), or if it's more like JVM debuggers that talk to the VM.

The NodeJS debugger specifically is like the JVM debugger. I have to launch node with --inspect={port number} and then connect to that port.

Also you can't have two debuggers attached to the same process and there is also no way to detach from a process while leaving it stopped.

Is it possible to detach and leave it running? That's the behavior I'm looking for. I launch my Go program with the debugger; it spawns node --inspect={port} script.js; delve hooks the fork-exec calls, sends an event, notices the process is not a Go process, and detaches, letting it run; the node process either runs normally (if I used --inspect) or - on its own - waits for the NodeJS debugger to attach (if I used --inspect-brk); my extension intercepts the process event, extracts the --inspect port from the cmdline (or checks for listening ports), and attaches the NodeJS debugger.

aarzilli · 2025-10-03T18:08:38Z

Ok, I think you can probably do that.

firelizzard18 · 2025-10-07T00:16:38Z

@aarzilli The real issue with non-Go processes is that TargetGroup.addTarget returned an error, which prevented the trap wait cleanup code from running (or prevented something important from running), which left the parent process in an unhappy, undebuggable state (or at least left delve in an unhappy state). I updated addTarget to return nil instead of the error when loading binary info fails.

I've resolved the issue with events by only sending a process spawned event for child processes. I am as certain as I can be that this will allow me to do what I want with heterogenous multiprocess debugging, so I'm marking it ready, though I can't properly test it until DAP-mode supports follow-exec (aka #4078).

aarzilli · 2025-10-07T12:43:48Z

pkg/proc/target_group.go

+			willFollow = false
+		}
+
+		// Notify listeners that a child process was spawned.


If you are doing this regardless just do it before the two previous ifs.

I want to include willFollow in the process spawned event. When I add the DAP start debugging request (which I plan to do in a separate PR), that probably needs to be conditional. If delve did not follow/attach to a child process, sending a start debugging request for that child probably won't work.

pkg/proc/target_group.go

aarzilli

Also, see test failures.

pkg/proc/target_group.go

firelizzard18 · 2025-10-08T18:54:08Z

pkg/proc/native/proc_linux.go

+				if errors.As(err, new(*proc.ErrBadBinaryInfo)) {
+					// If we are unable to load the binary info, log the error
+					// but don't return it. This allows the client to continue
+					// debugging the parent process.
+					logflags.DebuggerLogger().Warnf("Child target %d %q cannot be debugged: %v", pid, cmdline, err)
+				} else {
+					return nil, err
+				}


Continuing from #4171 (comment)

Ok, you are right but then the problem is that you are restarting the process.

I still don't see how this is different from the follow-exec-disabled case. The calls to Process.EntryPoint or BinaryInfo.LoadBinaryInfo are literally the only difference between the follow-exec-disabled and unable-to-load-binary-info paths (excluding read-only operations) so unless those function calls somehow cause the process to be restarted I don't see how that's possible. Unless the follow-exec-disabled case is also restarting the process, but if that's the case then it's not caused by my changes.

I was not able to reproduce the TestBadAttachRequest failure locally. Based on the stack traces of the exceptions, I am guessing the failure comes down to TargetGroup.addTarget being called from other places, so I moved the "log and ignore" code here. Which means it's platform specific. I found what I believe to be the equivalent code for Windows so I added a TODO, but I was unable to identify an equivalent block of code for darwin or freebsd. And I don't want to make changes that I can't test myself.

I see from TestFollowExec that follow-exec appears not to be implemented on darwin or freebsd, which explains why I couldn't find it.

I still don't see how this is different from the follow-exec-disabled case

It is the same but that path detaches from the child process, which restarts. Which means that by the time you see the process event that child will have made some progress executing.

Ah, are you saying "the process resumes execution and thus cannot be debugged from the entry point"? I misunderstood, I thought you were saying "the process is stopped and relaunched" or something along those lines. Though I suppose if the process hasn't executed any code there's not much of a difference.

Which means that by the time you see the process event that child will have made some progress executing.

I think this is just something I/we will have to live with. Having process events is better than not having process events, so I think this is worth implementing as-is. Maybe someone thinks up a better solution down the road. For my purposes, the child resuming is acceptable, because I need to use a non-ptrace debugger anyways, and if I use node's --inspect-brk flag then the part of the child I care about (the javascript, not the runtime) will be paused at entry anyways.

As far as handoff to another debugger, I found PTRACE_SEIZE which might be usable, but apparently it is "fragile and kernel-version dependent". Another thought that occurs to me is having delve act as a gdb rpc server and letting the caller connect that way. But that seems like something that needs a lot more planning before it would be worth considering.

firelizzard18 · 2025-10-10T18:10:54Z

Do you have any guesses why TestStepConcurrentDirect is failing? It passes locally and doesn't appear to use follow-exec. I put a breakpoint in the code I changed and it wasn't hit.

aarzilli · 2025-10-11T13:06:26Z

Do you have any guesses why TestStepConcurrentDirect is failing? It passes locally and doesn't appear to use follow-exec. I put a breakpoint in the code I changed and it wasn't hit.

Flake, probably. Where was it failing? TestGeneratedDoc is failing too.

firelizzard18 · 2025-10-12T20:36:10Z

It was on the aggregator/team city build for the previous commit.

aarzilli · 2025-10-15T13:34:57Z

It was on the aggregator/team city build for the previous commit.

That's not a TestGeneratedDoc failure.

firelizzard18 · 2025-10-15T16:43:50Z

That's not a TestGeneratedDoc failure.

Weren't we talking about TestStepConcurrentDirect? CI for a955866 failed on TestStepConcurrentDirect and CI for 24620f6 failed on TestGeneratedDoc.

derekparker · 2025-11-07T18:06:57Z

Is this ready for review now that #4078 is merged?

firelizzard18 · 2025-11-07T18:14:44Z

Yes, this is ready for review. As far as I can tell the CI failure is a flaky test.

derekparker · 2025-11-24T17:38:48Z

_fixtures/nongochild/child.bat

@@ -0,0 +1,2 @@
+@echo off
+echo Hello World


Can you add newline at end of script.

derekparker · 2025-11-24T17:38:52Z

_fixtures/nongochild/child.sh

@@ -0,0 +1,2 @@
+#!/bin/sh
+echo "Hello World"


Can you add newline at end of script.

derekparker · 2025-11-24T17:46:25Z

pkg/proc/target_group.go

 func (grp *TargetGroup) addTarget(p ProcessInternal, pid int, currentThread Thread, path string, stopReason StopReason, cmdline string) (*Target, error) {
 	logger := logflags.DebuggerLogger()
 	if len(grp.targets) > 0 {
+		willFollow := true


Why not initialize to grp.followExecEnabled?

Simply setting willFollow := grp.followExecEnabled didn't read right to me. Also, if there was some way for followExecEnabled to be false but still have followExecRegex nil, it would have printed out two "Detaching from child target" messages. I reworked it; hopefully it's more clear now.

firelizzard18

(Updated)

firelizzard18 · 2025-11-24T21:12:30Z

_fixtures/nongochild/child.bat

@@ -0,0 +1,2 @@
+@echo off
+echo Hello World


firelizzard18 · 2025-11-24T21:12:35Z

_fixtures/nongochild/child.sh

@@ -0,0 +1,2 @@
+#!/bin/sh
+echo "Hello World"


firelizzard18 · 2025-11-24T21:24:18Z

pkg/proc/target_group.go

 func (grp *TargetGroup) addTarget(p ProcessInternal, pid int, currentThread Thread, path string, stopReason StopReason, cmdline string) (*Target, error) {
 	logger := logflags.DebuggerLogger()
 	if len(grp.targets) > 0 {
+		willFollow := true


Simply setting willFollow := grp.followExecEnabled didn't read right to me. Also, if there was some way for followExecEnabled to be false but still have followExecRegex nil, it would have printed out two "Detaching from child target" messages. I reworked it; hopefully it's more clear now.

derekparker · 2025-11-25T19:13:19Z

Looks like there are still some failures around windows/arm64 and some capslock errors. You can fix the capslock errors by running go run _scripts/gen-capslock-all.go.

firelizzard18

TestLaunchWithFollowExec was failing (also on Linux, not sure why that didn't show up in CI). That test didn't exist when I started this PR; I assume that's why I didn't see the failure earlier. I rebased and force-pushed so I could test it. Fixing it on Linux was easy, I just added client.ExpectProcessEvent(t) to account for the new event. But on Windows something weird is happening (more details in a code comment).

firelizzard18 · 2025-11-26T20:37:59Z

service/dap/server_test.go

-		client.ExpectBreakpointEvent(t)
+		if runtime.GOOS != "windows" {
+			client.ExpectBreakpointEvent(t)
+		}
+		client.ExpectProcessEvent(t)


Adding ExpectProcessEvent is necessary because the spawned child process still generates a process event. But for some reason on Windows, the breakpoint doesn't happen and the process is terminated instead - I get a TerminatedEvent. Which is strange because I explicitly did not implement follow-exec process events on Windows, and this still happens even when I revert the changes to target_group.

This test is also failing in Windows on master:

$ go test -fullpath -run=^TestLaunchWithFollowExec$ -bench=- -json -count=1 -benchmem === RUN TestLaunchWithFollowExec DAP server listening at: [::]:58675 2025-11-26T14:41:25-08:00 warn layer=rpc Listening for remote connections (connections are not authenticated nor encrypted) Connecting to server at: [::]:58675 hello world c:/Users/firelizzard/Documents/delve/service/dap/server_test.go:733: got &dap.TerminatedEvent{Event:dap.Event{ProtocolMessage:dap.ProtocolMessage{Seq:0, Type:"event"}, Event:"terminated"}, Body:dap.TerminatedEventBody{Restart:json.RawMessage(nil)}}, want *dap.BreakpointEvent --- FAIL: TestLaunchWithFollowExec (1.59s) FAIL panic: Log in goroutine after TestLaunchWithFollowExec has completed: server stop triggered internally goroutine 12 [running]: testing.(*common).log(0xc000003a40, {0xc00136e1e0?, 0x21?}) C:/Program Files/Go/src/testing/testing.go:1030 +0x1ac testing.(*common).Log(0xc000003a40, {0xc000083fb0?, 0x0?, 0x0?}) C:/Program Files/Go/src/testing/testing.go:1180 +0x45 github.com/go-delve/delve/service/dap.startDAPServer.func1() c:/Users/firelizzard/Documents/delve/service/dap/server_test.go:127 +0x116 created by github.com/go-delve/delve/service/dap.startDAPServer in goroutine 8 c:/Users/firelizzard/Documents/delve/service/dap/server_test.go:119 +0x285 exit status 2 FAIL github.com/go-delve/delve/service/dap 2.887s

firelizzard18 · 2025-11-26T20:45:02Z

go run _scripts/gen-capslock-all.go didn't make any changes (git status shows now changes) when I ran it in my Windows VM, and when I ran it on my host (linux) the files all contain panic: got encoding/json/jsontext.Value, want variadic parameter of unnamed slice or string type (and a stack trace. Maybe rebasing onto master was enough.

derekparker · 2025-12-01T18:10:34Z

Hey, looks like there are still failures on the Windows builders:

server_test.go:736: got &dap.BreakpointEvent{Event:dap.Event{ProtocolMessage:dap.ProtocolMessage{Seq:0, Type:"event"}, Event:"breakpoint"}, Body:dap.BreakpointEventBody{Reason:"changed", Breakpoint:dap.Breakpoint{Id:1, Verified:true, Message:"", Source:(*dap.Source)(0x40002041b0), Line:6, Column:0, EndLine:0, EndColumn:0, InstructionReference:"", Offset:0}}}, want *dap.ProcessEvent

firelizzard18 · 2025-12-01T18:32:59Z

TestLaunchWithFollowExec is failing for me on the master branch (details) so I'm debugging blind. 6999ab9 passed on the one Windows runner that wasn't canceled and that's the version of the test that made sense, so I'll revert to that. But at this point I'm basically just using test-via-CI since it doesn't work when I test locally on Windows. Do I need to set up the environment somehow to make that work?

firelizzard18 changed the title ~~Process spawned event~~ pkg/proc: process spawned event Oct 2, 2025

firelizzard18 force-pushed the process-events branch 2 times, most recently from c877a70 to 9d5daaf Compare October 2, 2025 03:03

aarzilli reviewed Oct 2, 2025

View reviewed changes

pkg/proc/target_group.go Outdated Show resolved Hide resolved

service/dap/server.go Outdated Show resolved Hide resolved

aarzilli reviewed Oct 3, 2025

View reviewed changes

pkg/proc/target_group.go Outdated Show resolved Hide resolved

firelizzard18 marked this pull request as ready for review October 7, 2025 00:16

aarzilli reviewed Oct 7, 2025

View reviewed changes

aarzilli reviewed Oct 8, 2025

View reviewed changes

pkg/proc/target_group.go Outdated Show resolved Hide resolved

firelizzard18 commented Oct 8, 2025

View reviewed changes

Lslightly mentioned this pull request Oct 29, 2025

service/dap: add target command in console and support follow-exec in DAP interface #4078

Merged

derekparker reviewed Nov 24, 2025

View reviewed changes

firelizzard18 commented Nov 24, 2025

View reviewed changes

firelizzard18 force-pushed the process-events branch from e607841 to a2ffadb Compare November 26, 2025 20:20

firelizzard18 commented Nov 26, 2025

View reviewed changes

firelizzard18 added 6 commits November 26, 2025 14:10

pkg/proc: add an event for spawned processes

a2ffadb

Fix test

6999ab9

Fix?

4bd1f73

Events are still processed on Windows

6a87150

That skip was not necessary

1d5fb81

Update docs

bf7c970

Fix?

c46b1ac

		@@ -0,0 +1,2 @@
		@echo off
		echo Hello World No newline at end of file

		@@ -0,0 +1,2 @@
		#!/bin/sh
		echo "Hello World" No newline at end of file

pkg/proc: process spawned event #4171

Are you sure you want to change the base?

pkg/proc: process spawned event #4171

Uh oh!

Conversation

firelizzard18 commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

firelizzard18 commented Oct 2, 2025

Uh oh!

aarzilli commented Oct 2, 2025

Uh oh!

Uh oh!

Uh oh!

firelizzard18 commented Oct 2, 2025

Uh oh!

aarzilli commented Oct 3, 2025

Uh oh!

aarzilli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

firelizzard18 commented Oct 3, 2025

Uh oh!

aarzilli commented Oct 3, 2025

Uh oh!

firelizzard18 commented Oct 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aarzilli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

firelizzard18 Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

firelizzard18 commented Oct 10, 2025

Uh oh!

aarzilli commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

firelizzard18 commented Oct 12, 2025

Uh oh!

aarzilli commented Oct 15, 2025

Uh oh!

firelizzard18 commented Oct 15, 2025

Uh oh!

derekparker commented Nov 7, 2025

Uh oh!

firelizzard18 commented Nov 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

firelizzard18 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

firelizzard18 commented Oct 2, 2025 •

edited

Loading

firelizzard18 Oct 8, 2025 •

edited

Loading

aarzilli commented Oct 11, 2025 •

edited

Loading